kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
9c95b24927
TensorRT-LLMs/_cpp_gen/runtime.html
石晓伟 9c95b24927
update github pages (#540)
2023-12-04 16:26:13 +08:00

6447 lines
1.3 MiB
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Runtime &mdash; tensorrt_llm documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=19f00094" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=888ff710"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token." href="../blogs/H100vsA100.html" />
<link rel="prev" title="Runtime" href="../python-api/tensorrt_llm.runtime.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
tensorrt_llm
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">TensorRT-LLM Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../gpt_runtime.html">C++ GPT Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../batch_manager.html">The Batch Manager in TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../gpt_attention.html">Multi-head, Multi-query and Group-query Attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="../precision.html">Numerical Precision</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance.html">Performance of TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../2023-05-19-how-to-debug.html">How to debug</a></li>
<li class="toctree-l1"><a class="reference internal" href="../2023-05-17-how-to-add-a-new-model.html">How to add a new model</a></li>
<li class="toctree-l1"><a class="reference internal" href="../graph-rewriting.html">Graph Rewriting Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../memory.html">Memory Usage of TensorRT-LLM</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Python API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.layers.html">Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.functional.html">Functionals</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.models.html">Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.plugin.html">Plugin</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.quantization.html">Quantization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.runtime.html">Runtime</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">C++ API</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Runtime</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#buffermanager-h">bufferManager.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv412tensorrt_llm"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#common-h">common.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SizeType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TokenIdType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::StringPtrMap</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#cudaevent-h">cudaEvent.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::pointer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventEj"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::CudaEvent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::CudaEvent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent3getEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::get()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::synchronize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::element_type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::EventPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent6mEventE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::mEvent</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaEvent::Deleter</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#cudastream-h">cudaStream.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamEji"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::CudaStream()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::CudaStream()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream9getDeviceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::getDevice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream3getEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::get()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream11synchronizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::synchronize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::record()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::record()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::wait()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::wait()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::StreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mStreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::mStream</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mDeviceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::mDevice</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::CudaStream::Deleter</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#decodinginput-h">decodingInput.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType8SizeType9TensorPtr9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::DecodingInput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput4stepE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::step</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9maxLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::maxLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput16maxKvCacheLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::maxKvCacheLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9batchSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::batchSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6logitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::logits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6endIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::endIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput8finishedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::finished</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::sequenceLimitLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::embeddingBias</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput7lengthsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::lengths</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput12badWordsListE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::badWordsList</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13stopWordsListE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::stopWordsList</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::noRepeatNgramSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingInput::cacheIndirection</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#decodingoutput-h">decodingOutput.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::DecodingOutput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput3idsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::ids</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14newTokensStepsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::newTokensSteps</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9newTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::newTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput12newTokensVecE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::newTokensVec</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8finishedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::finished</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11finishedSumE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::finishedSum</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8logProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::logProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::cumLogProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9parentIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::parentIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput7lengthsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::lengths</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::cacheIndirection</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::beamHypotheses</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::kNegativeInfinity</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DecodingOutput::BeamHypotheses</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#generationinput-h">generationInput.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput22GenericGenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::GenericGenerationInput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5endIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::endId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5padIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::padId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput3idsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::ids</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput7lengthsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::lengths</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput6packedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::packed</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13embeddingBiasE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::embeddingBias</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12badWordsListE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::badWordsList</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13stopWordsListE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::stopWordsList</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12maxNewTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::maxNewTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput18promptTuningParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationInput::promptTuningParams</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationInput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput4BaseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationInput::Base</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationInput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationInput::GenerationInput()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#generationoutput-h">generationOutput.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::Callback</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput23GenericGenerationOutputE9TensorPtr9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::GenericGenerationOutput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput3idsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::ids</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput7lengthsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::lengths</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput11cumLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::cumLogProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8logProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::logProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput13contextLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::contextLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16generationLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::generationLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16onTokenGeneratedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericGenerationOutput::onTokenGenerated</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationOutput</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput4BaseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationOutput::Base</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationOutput::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenerationOutput::GenerationOutput()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#gptdecoder-h">gptDecoder.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm6layersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::layers</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::~IGptDecoder()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::setup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::forward()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::gatherTree()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder17getSamplingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::getSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder22acceptDraftTokensByIdsERK7ITensorRK7ITensorRK7ITensorRK7ITensorR7ITensorRK7ITensorR7ITensorR7ITensorRKN13BufferManager13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::acceptDraftTokensByIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder25acceptDraftTokensByLogitsER7ITensorRK7ITensorR7ITensorR7ITensorRK7ITensorR7ITensor8SizeType8SizeTypebfP13curandState_tRKN13BufferManager13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::acceptDraftTokensByLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoder::create()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::GptDecoder()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::setup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::forward()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::gatherTree()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder17getSamplingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::getSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder8mManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::mManager</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10mAllocatorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::mAllocator</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::mDynamicDecodeLayer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder14mLogProbsTiledE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::mLogProbsTiled</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder15mSamplingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoder::mSamplingConfig</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#gptdecoderbatch-h">gptDecoderBatch.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatchE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::GptDecoderBatch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::setup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::newRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::newBatch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::forwardSync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::forwardSync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getFinished()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getOutputIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getOutputIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::finalize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::finalize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getParentIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getCumLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getCumLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch15getAllNewTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getAllNewTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getNewTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getNbSteps()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::getNbFinished()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::GptDecoderPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::DecodingInputPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::DecodingOutputPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::postProcessRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mVocabSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mVocabSizePadded</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mStream</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mBufferManager</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mForwardToken</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mForwardEvent</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mStreams</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDecoders</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDecodingInputs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDecodingOutputs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingInput</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingOutput</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mDraftTokenIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDraftTokenIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mDraftLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDraftLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mAcceptByLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mAcceptByLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mNumDraftTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mNumDraftTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mCurandStatesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mCurandStates</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mNbSteps</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mFinished</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mFinishedSum</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mMaxNewTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mBeamWidths</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch23mGeneratedTokensPerStepE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mGeneratedTokensPerStep</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mFinishedStepsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mFinishedSteps</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mDraftProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mDraftProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mTargetProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mTargetProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mMaxSequenceLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxKvCacheLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mMaxKvCacheLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mActualBatchSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxTokensPerStepE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptDecoderBatch::mMaxTokensPerStep</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#gptjsonconfig-h">gptJsonConfig.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::GptJsonConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getModelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getName()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getPrecision()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getTensorParallelism()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getPipelineParallelism()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::getWorldSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::engineFilename()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::engineFilename()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::parse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::parse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::parse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5mNameE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::mName</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::mPrecision</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::mTensorParallelism</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::mPipelineParallelism</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptJsonConfig::mGptModelConfig</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#gptmodelconfig-h">gptModelConfig.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::ModelVariant</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::GptModelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getVocabSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getVocabSizePadded()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getNbLayers()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getNbHeads()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getNbKvHeads()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setNbKvHeads()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getHiddenSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getSizePerHead()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::useGptAttentionPlugin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::useGptAttentionPlugin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::usePackedInput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::usePackedInput()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::usePagedKvCache()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::usePagedKvCache()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getTokensPerBlock()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setTokensPerBlock()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getQuantMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setQuantMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::supportsInflightBatching()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxBatchSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxBatchSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxInputLen()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxInputLen()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxOutputLen()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxOutputLen()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxNumTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxNumTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePromptTuningEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::usePromptTuning()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig30getMaxPromptEmbeddingTableSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxPromptEmbeddingTableSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig30setMaxPromptEmbeddingTableSizeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxPromptEmbeddingTableSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::computeContextLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::computeContextLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::computeGenerationLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::computeGenerationLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getModelVariant()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setModelVariant()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::useCustomAllReduce()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::useCustomAllReduce()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxDraftLenE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::setMaxDraftLen()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig19getMaxTokensPerStepEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::getMaxTokensPerStep()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mVocabSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mNbLayers</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mNbHeads</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mNbKvHeads</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mHiddenSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mDataType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mUseGptAttentionPlugin</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mInputPacked</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mPagedKvCache</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mTokensPerBlock</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mQuantMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxBatchSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxInputLen</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxOutputLen</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxNumTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mComputeContextLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig24mComputeGenerationLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mComputeGenerationLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mModelVariant</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mUseCustomAllReduce</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig28mMaxPromptEmbeddingTableSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxPromptEmbeddingTableSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxDraftLenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptModelConfig::mMaxDraftLen</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#gptsession-h">gptSession.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::batch_manager</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::batch_manager::kv_cache_manager</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSessionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::LoggerPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::GptSession()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::GptSession()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::GptSession()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getLoggerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::getLogger()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::getBufferManager()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getModelConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::getModelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::getWorldConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getDeviceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::getDevice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::generate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::KvCacheManager</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13KvCacheConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::KvCacheConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::TokenGeneratedCallback</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13useCudaGraphsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::useCudaGraphs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15generateBatchedERNSt6vectorI16GenerationOutputEERKNSt6vectorI15GenerationInputEERK14SamplingConfigRK22TokenGeneratedCallback"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::generateBatched()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession5setupERK6Config"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::setup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType8SizeTypeb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createContexts()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createBuffers()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createDecoders()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession20createKvCacheManagerE8SizeType8SizeType8SizeType8SizeTypeRK13KvCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createKvCacheManager()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createCustomAllReduceWorkspace()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession18executeContextStepERKNSt6vectorI15GenerationInputEERKNSt6vectorI8SizeTypeEEPK14KvCacheManager"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::executeContextStep()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession21executeGenerationStepE8SizeTypeRKNSt6vectorI15GenerationInputEERNSt6vectorI16GenerationOutputEERKNSt6vectorI8SizeTypeEEP14KvCacheManagerRNSt6vectorIbEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::executeGenerationStep()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16decoderStepAsyncE8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::decoderStepAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::shouldStopSync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8finalizeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::finalize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::kvCacheAddSequences()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession11initDecoderER7ITensorRK15GenerationInputRK16GenerationOutputRK14SamplingConfig8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::initDecoder()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession30createOnTokenGeneratedCallbackER16GenerationOutput"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::createOnTokenGeneratedCallback()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mModelConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mModelConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mWorldConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mWorldConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mDeviceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mDevice</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13mPipelineCommE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mPipelineComm</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession11mCommStreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mCommStream</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10mCommEventE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mCommEvent</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9mCommPtrsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mCommPtrs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17mIpcMemoryHandlesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mIpcMemoryHandles</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mDecoderMaxSequenceLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession24mDecoderMaxKvCacheLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mDecoderMaxKvCacheLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mLoggerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mLogger</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mRuntimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mRuntime</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15mKvCacheManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mKvCacheManager</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17mMicroBatchConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mMicroBatchConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9mDecodersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mDecoders</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mBuffersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mBuffers</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15mReceivedEventsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mReceivedEvents</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mCudaGraphMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::mCudaGraphInstances</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::Config</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::CudaGraphExecutor</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GptSession::MicroBatchConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime5utilsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::utils</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::utils::loadEngine()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#ibuffer-h">iBuffer.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE"><code class="docutils literal notranslate"><span class="pre">MemoryTypeString::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4nameE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4sizeE"><code class="docutils literal notranslate"><span class="pre">DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE"><code class="docutils literal notranslate"><span class="pre">TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PointerElementType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryType::kGPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryType::kCPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryType::kPINNED</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::constPointerCast()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::constPointerCast()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::bufferCast()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::bufferCast()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryTypeString</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PhonyNameDueToError::value</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime14DataTypeTraitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DataTypeTraits</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PhonyNameDueToError::type</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4nameE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PhonyNameDueToError::name</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4sizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PhonyNameDueToError::size</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DataTypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DataTypeTraits::type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4nameE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DataTypeTraits::name</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4sizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::DataTypeTraits::size</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::BufferDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::operator</span> <span class="pre">nvinfer1::DataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::getDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType9isPointerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::isPointer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::isUnsigned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType7getSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::getSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::kTrtPointerType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mDataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::mDataType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mUnsignedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::mUnsigned</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType8mPointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferDataType::mPointer</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0_bEN12tensorrt_llm7runtime11TRTDataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIbEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::value</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TRTDataType::kUnderlyingType</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::UniquePtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::SharedPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::UniqueConstPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::SharedConstPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::DataType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::data()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::data()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::data()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::data()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7getSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getSizeInBytes()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getCapacityEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getCapacity()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer15getDataTypeNameEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getDataTypeName()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getMemoryType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer17getMemoryTypeNameEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::getMemoryTypeName()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::resize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7releaseEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::release()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::~IBuffer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::IBuffer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferaSERK7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::memoryType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::IBuffer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IBuffer::toBytes()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::value_type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::size_type</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::reference</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::const_reference</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::pointer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::const_pointer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::iterator</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::const_iterator</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::BufferRange()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5beginEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::begin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange3endEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::end()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange5beginEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::begin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange3endEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::end()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange6cbeginEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::cbegin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange4cendEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::cend()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange6cbeginEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::cbegin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4cendEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::cend()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4sizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::size()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRangeixE9size_type"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::operator[]()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRangeixE9size_type"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::operator[]()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::mData</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::BufferRange::mSize</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#igptdecoderbatch-h">iGptDecoderBatch.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::TokenPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::newRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::forwardSync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::forward()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getOutputIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch8finalizeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::finalize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getFinished()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getCumLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getCumLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getParentIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::getNbSteps()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IGptDecoderBatch::IGptDecoderBatch()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder_batch</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder_batch::Output</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder_batch::Request</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder_batch::Input</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder_batch::Token</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#istatefulgptdecoder-h">iStatefulGptDecoder.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::setup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::newBatch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::forwardAsync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder11forwardSyncEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::forwardSync()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::forward()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder8finalizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::finalize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getOutputIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder14getCumLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getCumLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder11getLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getNewTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder15getAllNewTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getAllNewTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::getNbFinished()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::~IStatefulGptDecoder()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IStatefulGptDecoder::IStatefulGptDecoder()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder::Input</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::decoder::Output</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#itensor-h">iTensor.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv48nvinfer1"><code class="docutils literal notranslate"><span class="pre">nvinfer1</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::UniquePtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::SharedPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::UniqueConstPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::SharedConstPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::Shape</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7DimTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::DimType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::~ITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor8getShapeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::getShape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::reshape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor6resizeENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::resize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::ITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensoraSERK7ITensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::squeeze()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::unsqueeze()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::shapeEquals()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERKNSt16initializer_listI8SizeTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::shapeEquals()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEbPK1T8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::shapeEquals()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor6volumeERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::volume()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::volumeNonNegative()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::squeeze()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeERK5Shape8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::unsqueeze()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::slice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::view()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::wrap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::makeShape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor8toStringERK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::toString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapeRK5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::shapeEquals()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsEbRK5ShapePK1T8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::shapeEquals()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::ITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor8castSizeE6size_t"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::ITensor::castSize()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#ipcutils-h">ipcUtils.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::setPeerAccess()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::IpcMemory()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::~IpcMemory()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::getCommPtrsTensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::FLAGS_SIZE</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::allocateIpcMemory()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::destroyIpcMemory()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::mWorldConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::mCommPtrs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::mBufferSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::IpcMemory::mBufferPtr</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#memorycounters-h">memoryCounters.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::SizeType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::DiffType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::MemoryCounters()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getGpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getCpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getGpuDiff()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getCpuDiff()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getPinnedDiff()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateEv8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::allocate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::allocate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateEv8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::deallocate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::deallocate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters8toStringEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::toString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::getInstance()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::bytesToString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::bytesToString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mGpuE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mGpu</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mCpuE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mCpu</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters7mPinnedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mPinned</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mGpuDiff</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mCpuDiff</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mPinnedDiff</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters9mInstanceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::MemoryCounters::mInstance</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#samplingconfig-h">samplingConfig.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::SamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9beamWidthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::beamWidth</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig11temperatureE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::temperature</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9minLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::minLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::repetitionPenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::presencePenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topKE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::topK</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::topP</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig10randomSeedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::randomSeed</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9topPDecayE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::topPDecay</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig7topPMinE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::topPMin</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::topPResetIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::beamSearchDiversityRate</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::lengthPenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig24draftAcceptanceThresholdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::draftAcceptanceThreshold</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::FloatType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::SamplingConfig::OptVec</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#tllmlogger-h">tllmLogger.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10TllmLoggerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TllmLogger</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TllmLogger::log()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8getLevelEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TllmLogger::getLevel()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::TllmLogger::setLevel()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#worldconfig-h">worldConfig.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::WorldConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getTensorParallelism()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::isTensorParallel()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getPipelineParallelism()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::isPipelineParallel()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getGpusPerNode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getDevice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getPipelineParallelRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getTensorParallelRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::isFirstPipelineParallelRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::isLastPipelineParallelRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig11getLastRankEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getLastRank()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::getPipelineParallelGroup()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::validConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mpi()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mpi()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mTensorParallelism</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mPipelineParallelism</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig5mRankE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mRank</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::WorldConfig::mGpusPerNode</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#prompttuningparams-h">promptTuningParams.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::SizeType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams25GenericPromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::GenericPromptTuningParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams14embeddingTableE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::embeddingTable</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams5tasksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::tasks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9vocabSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::vocabSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams19promptTuningEnabledE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::GenericPromptTuningParams::promptTuningEnabled</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PromptTuningParams</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PromptTuningParams::TensorPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PromptTuningParams::SizeType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams18PromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PromptTuningParams::PromptTuningParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams15fillTasksTensorE9TensorPtrK8SizeTypeK8SizeTypeRKNSt6vectorI8SizeTypeEERKNSt6vectorI8SizeTypeEERK13BufferManagerb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime::PromptTuningParams::fillTasksTensor()</span></code></a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Blogs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H100vsA100.html">H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token.</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H200launch.html">H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">tensorrt_llm</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Runtime</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/_cpp_gen/runtime.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="runtime">
<h1>Runtime<a class="headerlink" href="#runtime" title="Link to this heading"></a></h1>
<section id="buffermanager-h">
<h2>bufferManager.h<a class="headerlink" href="#buffermanager-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv412tensorrt_llm">
<span id="_CPPv312tensorrt_llm"></span><span id="_CPPv212tensorrt_llm"></span><span id="tensorrt_llm"></span><span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><a class="headerlink" href="#_CPPv412tensorrt_llm" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimeE">
<span id="_CPPv3N12tensorrt_llm7runtimeE"></span><span id="_CPPv2N12tensorrt_llm7runtimeE"></span><span id="tensorrt_llm::runtime"></span><span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimeE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManagerE"></span><span id="tensorrt_llm::runtime::BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;bufferManager.h&gt;</em></div>
<p>A helper class for managing memory on host and device. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10IBufferPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10IBufferPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0bd5b422b1e3a5d2171f66532f27f413"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IBufferPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10ITensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10ITensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a5f8cccf8f635cf3cfbec353e1a7317b9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ITensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ae977afd5a423417ac57f3bffa8162b33"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::BufferManager::BufferManager__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ac641ea2cdcc436ed29ce3bacc42d549a"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferManager</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Construct a <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1BufferManager"><span class="std std-ref">BufferManager</span></a>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>cudaStream</strong> – <strong>[in]</strong> The cuda stream to use for all operations on GPU (allocation, de-allocation, copying, etc.). </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::gpu__std::s.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a2cf0e00fd6ac0cfe8b593749583a7a03"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the GPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::gpu__nvinfer1::Dims.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a5a023b2d5f629bc86e262f9733a12a59"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the GPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::allocate__MemoryType.std::s.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1bdafec3fa633fdb0d321f5e748b3174"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size and memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::allocate__MemoryType.nvinfer1::Dims.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0e4e4c0e248e95e19061a61fadec30cc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions and memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::emptyBuffer__MemoryType.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0e78e3aa2aa1d635ae230996b449f9a4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyBuffer</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Create an empty <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given memory type. It may be resized later. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::emptyTensor__MemoryType.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1d0157fa19df03ba561d15d5b64de259"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyTensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Create an empty <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given memory type. It may be reshaped later. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::setZero__IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1afe66c31bfc8e00ebbdb8405194be0db9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setZero</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Set the contents of the given <code class="docutils literal notranslate"><span class="pre">buffer</span></code> to zero. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copy__voidCP.IBufferR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1adc12c1388e8a3392ac39a3ae64d518"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">srcType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.voidP.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a313f2bd39b26e568c97f092a98ac20f9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dst</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dstType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::copy__voidCP.IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3086524a384108c45199076f536d4d9c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.voidPC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a583c0a02dcd51050b5b4273a19c0bde7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a406d3f69a661d5a3bcb5dded51626c44"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copyFrom__IBufferCR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3101d71e18fa84154b687236487dfc5f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copyFrom__ITensorCR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a80a336bdfbd0faf2d76e470354bcddf0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a51f44f1cf97f9c0b686048d37291ea5a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromEP1TN8nvinfer14DimsE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromEP1TN8nvinfer14DimsE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ac42305510a739c47d3c4358b97560f3e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a8feac5727a72ee0d1c50b3ff5ee4e979"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType" title="Link to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager9getStreamEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager9getStreamEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager9getStreamEv"></span><span id="tensorrt_llm::runtime::BufferManager::getStreamC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aa8eb6eafae45f681f082cab5daa26029"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getStream</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager9getStreamEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the underlying cuda stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEv"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolReservedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a92a21be0a954deb04c9c6e8540a2141b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolReserved</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEv" title="Link to this definition"></a><br /></dt>
<dd><p>The current size of the memory reserved by the memory pool. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEv"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolUsedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a73ae2808af584ba85c1a61134f025659"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolUsed</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEv" title="Link to this definition"></a><br /></dt>
<dd><p>The current size of the memory used by the memory pool. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEv"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolFreeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a822c7eaa19d0bdebd460fdc4cf6ee82a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolFree</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEv" title="Link to this definition"></a><br /></dt>
<dd><p>The current size of the memory free in the memory pool. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToENSt6size_tE"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolTrimTo__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a5e74ccf0e2e2132ae5834c31209f87d3"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolTrimTo</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Try to trim the memory reserved by the pool to <code class="docutils literal notranslate"><span class="pre">size</span></code> bytes. This synchronizes implicitly with the stream. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::cpu__std::s.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a7edb3cdde9b85c01fec18f3f4f34282f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::cpu__nvinfer1::Dims.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3beb40d9ddabd96774167df148efc676"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::pinned__std::s.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a6a58588cfc27c05c1d7dbb2f043672d3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates a pinned <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::pinned__nvinfer1::Dims.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3cfc4376c5e62876586e1e07e08d3069"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Allocates a pinned <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the CPU. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE"></span><span id="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE__auto"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1af4254e3a9525a8051face3c7ca729b65"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBYTE_TYPE</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager7mStreamE"></span><span id="tensorrt_llm::runtime::BufferManager::mStream__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aeab688938251784e88b66cff01a0dde3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager7mStreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi"></span><span id="tensorrt_llm::runtime::BufferManager::initMemoryPool__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aba8fe4df93bbf829d0cefa8bcc63e657"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">initMemoryPool</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEi">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEi"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEi"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolReserved__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a76510e249e79d5ad43ca5cccf587cbec"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolReserved</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager18memoryPoolReservedEi" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEi">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEi"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEi"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolUsed__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a32130ebd3537d477a5002edfacd19058"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolUsed</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager14memoryPoolUsedEi" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEi">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEi"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEi"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolFree__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a198cac633154050fbb500d3abae689d7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolFree</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager14memoryPoolFreeEi" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToEiNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToEiNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToEiNSt6size_tE"></span><span id="tensorrt_llm::runtime::BufferManager::memoryPoolTrimTo__i.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a9702f3969f11fa908f4b24662bd8ff63"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryPoolTrimTo</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager16memoryPoolTrimToEiNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="common-h">
<h2>common.h<a class="headerlink" href="#common-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime8SizeTypeE"></span><span class="target" id="common_8h_1a8da5a8de61b5f096333bdd15d5a16875"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime11TokenIdTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TokenIdTypeE"></span><span class="target" id="common_8h_1a1de916f1c3f3c8d1f9c66320afc6df17"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenIdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime12StringPtrMapE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime12StringPtrMapE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="common_8h_1a43946c471b82feb36a6350de9cde277d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StringPtrMap</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unordered_map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE" title="tensorrt_llm::runtime::StringPtrMap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</section>
<section id="cudaevent-h">
<h2>cudaEvent.h<a class="headerlink" href="#cudaevent-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEventE"></span><span id="tensorrt_llm::runtime::CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a765323cdb24844d31c8f9e354a5194f8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaEvent_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventEj">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent9CudaEventEj"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent9CudaEventEj"></span><span id="tensorrt_llm::runtime::CudaEvent::CudaEvent__unsigned-i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1aae9d65fd3cc3d42763c1219710dcd6ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">flags</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaEventDisableTiming</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventEj" title="Link to this definition"></a><br /></dt>
<dd><p>Creates a new cuda event. The event will be destroyed in the destructor.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>flags</strong> – Flags for event creation. By default, event timing is disabled. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb"></span><span id="tensorrt_llm::runtime::CudaEvent::CudaEvent__pointer.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a8ac4cb3f5ac924e72862c1c5fd033cbd"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsEvent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb" title="Link to this definition"></a><br /></dt>
<dd><p>Pass an existing cuda event to this object.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>event</strong> – The event to pass to this object. </p></li>
<li><p><strong>ownsEvent</strong> – Whether this object owns the event and destroys it in the destructor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent3getEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent3getEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent3getEv"></span><span id="tensorrt_llm::runtime::CudaEvent::getC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a530c756fe9e6ad149b813659b3644f16"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent3getEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the event associated with this object. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv"></span><span id="tensorrt_llm::runtime::CudaEvent::synchronizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a9216cdfafea99849fb47b1be60fb1a7e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">synchronize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Synchronizes the event. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent12element_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent12element_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a411326cde2f9f947c5d92abe724ebb34"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">element_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_pointer_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent8EventPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent8EventPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1ac5468fc310bab51276ed6d6212e7240f"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">EventPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE" title="tensorrt_llm::runtime::CudaEvent::element_type"><span class="n"><span class="pre">element_type</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE" title="tensorrt_llm::runtime::CudaEvent::Deleter"><span class="n"><span class="pre">Deleter</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent6mEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent6mEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent6mEventE"></span><span id="tensorrt_llm::runtime::CudaEvent::mEvent__EventPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a3231de6317de1857e5f699a94d59dcf2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE" title="tensorrt_llm::runtime::CudaEvent::EventPtr"><span class="n"><span class="pre">EventPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent6mEventE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7DeleterE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7DeleterE"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::Deleter__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a7807c6a0240c9d01f09dbde19b422d05"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsEvent</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1aec3a7bdb695dc6598d6c6b8c5cbd4598"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::call-operator__pointerCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a82af09350919fceb1f0f4c43d1822d37"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">()</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::mOwnsEvent__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a4df7640a97930eaf43a1ca952f3f919f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOwnsEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="cudastream-h">
<h2>cudaStream.h<a class="headerlink" href="#cudastream-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStreamE"></span><span id="tensorrt_llm::runtime::CudaStream"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamEji">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream10CudaStreamEji"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream10CudaStreamEji"></span><span id="tensorrt_llm::runtime::CudaStream::CudaStream__unsigned-i.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a2ba13a61587813c68c018a64ed2967fe"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">flags</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaStreamNonBlocking</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamEji" title="Link to this definition"></a><br /></dt>
<dd><p>Creates a new cuda stream on the current device. The stream will be destroyed in the destructor.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>flags</strong> – Flags for stream creation. See ::cudaStreamCreateWithFlags for a list of valid flags that can be passed. </p></li>
<li><p><strong>priority</strong> – Priority of the stream. Lower numbers represent higher priorities. See ::cudaDeviceGetStreamPriorityRange for more information about the meaningful stream priorities that can be passed. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib"></span><span id="tensorrt_llm::runtime::CudaStream::CudaStream__cudaStream_t.i.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a4ff09ea22fc6679e2d93b772e148d19e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsStream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib" title="Link to this definition"></a><br /></dt>
<dd><p>Pass an existing cuda stream to this object.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>stream</strong> – The stream to pass to this object. </p></li>
<li><p><strong>device</strong> – The device on which the stream was created. </p></li>
<li><p><strong>ownsStream</strong> – Whether this object owns the stream and destroys it in the destructor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream9getDeviceEv"></span><span id="tensorrt_llm::runtime::CudaStream::getDeviceC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a2209ae12d3b5a27740d66bec35e686c9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream9getDeviceEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the device on which the stream was created. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream3getEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream3getEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream3getEv"></span><span id="tensorrt_llm::runtime::CudaStream::getC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a9ba739eedb71b18bc4a58ff480c82a27"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream3getEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the stream associated with this object. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream11synchronizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream11synchronizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream11synchronizeEv"></span><span id="tensorrt_llm::runtime::CudaStream::synchronizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a3e3def9cc0e09e9724e9e68232ed2679"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">synchronize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream11synchronizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Synchronizes the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE"></span><span id="tensorrt_llm::runtime::CudaStream::record__CudaEvent::pointerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1acc03555968e2361dc08d28f2228a1e31"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">record</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE" title="Link to this definition"></a><br /></dt>
<dd><p>Record an event on the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent"></span><span id="tensorrt_llm::runtime::CudaStream::record__CudaEventCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1ace6075f1266bdefe0b2033717dd8b14a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">record</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent" title="Link to this definition"></a><br /></dt>
<dd><p>Record an event on the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE"></span><span id="tensorrt_llm::runtime::CudaStream::wait__CudaEvent::pointerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a97622ff55b119c5eec1b096115462098"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wait</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for an event. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent"></span><span id="tensorrt_llm::runtime::CudaStream::wait__CudaEventCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a60cfbc942ee8d5f787ae66f891d91766"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wait</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for an event. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream9StreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream9StreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a42e6eb53f179659c51fbd8981383af64"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_pointer_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">cudaStream_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE" title="tensorrt_llm::runtime::CudaStream::Deleter"><span class="n"><span class="pre">Deleter</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7mStreamE"></span><span id="tensorrt_llm::runtime::CudaStream::mStream__StreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a21588e234ee724b69f92d1ef26779334"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE" title="tensorrt_llm::runtime::CudaStream::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mStreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7mDeviceE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7mDeviceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7mDeviceE"></span><span id="tensorrt_llm::runtime::CudaStream::mDevice__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a5dc4aea4b9abfa576f166ce152b675e0"></span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDevice</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mDeviceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7DeleterE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7DeleterE"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::Deleter__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1aa03f43a2e696a0433dd1a1c14ce129c2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsStream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1adae80ccd16ccabd753f118cdb4111a51"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::call-operator__cudaStream_tCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1a077db80f4a8d30a5f2e0adde1ec6372a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">()</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::mOwnsStream__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1a71b80a1570697e35d7b5edb51904d9bd"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOwnsStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="decodinginput-h">
<h2>decodingInput.h<a class="headerlink" href="#decodinginput-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInputE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInputE"></span><span id="tensorrt_llm::runtime::DecodingInput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a8e97b9538b8259558b0c15cbe2a59d15"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType8SizeType9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType8SizeType9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType8SizeType9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::DecodingInput::DecodingInput__SizeType.SizeType.SizeType.TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a486752fda6f648f8d9238df036673c91"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxKvCacheLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType8SizeType9TensorPtr9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput4stepE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput4stepE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput4stepE"></span><span id="tensorrt_llm::runtime::DecodingInput::step__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a82755273657729ca89f9186abeeb713c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">step</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput4stepE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9maxLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9maxLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9maxLengthE"></span><span id="tensorrt_llm::runtime::DecodingInput::maxLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1ac4e6e64f43c3994f785a7047afd4578e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9maxLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput16maxKvCacheLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput16maxKvCacheLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput16maxKvCacheLengthE"></span><span id="tensorrt_llm::runtime::DecodingInput::maxKvCacheLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a18ff148f178c5f1aa7d32a0fa8a8dca8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxKvCacheLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput16maxKvCacheLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9batchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9batchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9batchSizeE"></span><span id="tensorrt_llm::runtime::DecodingInput::batchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a5d6979ade3dcff577fcf46c70c887917"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">batchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9batchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput6logitsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput6logitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput6logitsE"></span><span id="tensorrt_llm::runtime::DecodingInput::logits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1aafadc431e8d06615d8425d82b1afaaa8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6logitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput6endIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput6endIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput6endIdsE"></span><span id="tensorrt_llm::runtime::DecodingInput::endIds__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6bb630897ccbb4f437cf9ec517912d25"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6endIdsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput8finishedE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput8finishedE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput8finishedE"></span><span id="tensorrt_llm::runtime::DecodingInput::finished__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a8538e4b4b0f7c4e9cf495fba7fb9e648"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finished</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput8finishedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE"></span><span id="tensorrt_llm::runtime::DecodingInput::sequenceLimitLength__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1aa14e54120b58626beca8705ced84a79c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLimitLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE"></span><span id="tensorrt_llm::runtime::DecodingInput::embeddingBias__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a132486f973befa6c1c54bb0c22aab289"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBias</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput7lengthsE"></span><span id="tensorrt_llm::runtime::DecodingInput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1ad44428022172d330f631c81a712ce206"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput7lengthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput12badWordsListE"></span><span id="tensorrt_llm::runtime::DecodingInput::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1acfc2ebcd8202fc2330466a8773b1ab6d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput12badWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13stopWordsListE"></span><span id="tensorrt_llm::runtime::DecodingInput::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6979dbc3a119fefd9f50c2132c1c95e7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13stopWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE"></span><span id="tensorrt_llm::runtime::DecodingInput::noRepeatNgramSize__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a764f889cfac79873d7c88978e9d42df4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">noRepeatNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::DecodingInput::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6eccbda69f8c35e1eda57e9eb24ca930"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="decodingoutput-h">
<h2>decodingOutput.h<a class="headerlink" href="#decodingoutput-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutputE"></span><span id="tensorrt_llm::runtime::DecodingOutput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1aeb215aae60278c44bcdbd17ae5f7c8e1"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr"></span><span id="tensorrt_llm::runtime::DecodingOutput::DecodingOutput__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a33bcc26e94af9077c4c8d1ba65acd39c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput3idsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a9937790aaf8c5cbb5230236afc7656be"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput3idsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14newTokensStepsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14newTokensStepsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14newTokensStepsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::newTokensSteps__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a6e42e3b0fd2b24ba8a04f78f837b207a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newTokensSteps</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14newTokensStepsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9newTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9newTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9newTokensE"></span><span id="tensorrt_llm::runtime::DecodingOutput::newTokens__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a1c92d23772ed90b78f9ac8c86f46d9fd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9newTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput12newTokensVecE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput12newTokensVecE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput12newTokensVecE"></span><span id="tensorrt_llm::runtime::DecodingOutput::newTokensVec__std::vector:TensorPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a0df2485f70ab5959e706840f7635e4c3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newTokensVec</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput12newTokensVecE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput8finishedE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput8finishedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput8finishedE"></span><span id="tensorrt_llm::runtime::DecodingOutput::finished__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a721bfb18d093d881f2f3a7d288dfb0e4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finished</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8finishedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput11finishedSumE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput11finishedSumE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput11finishedSumE"></span><span id="tensorrt_llm::runtime::DecodingOutput::finishedSum__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a2bb2c2cc930598e59363a8d61e2f0fc3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finishedSum</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11finishedSumE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput8logProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a4abf75dc398349212b15b93f843fc03c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8logProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::cumLogProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a428f7d981f0617bda37e6f64bb4f0007"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9parentIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9parentIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9parentIdsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::parentIds__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a9f518ec59bd0df527432074008d974a6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parentIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9parentIdsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput7lengthsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a7425f953e704cffb2b917d475fc90c12"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput7lengthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::DecodingOutput::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a6ee6dfc4bf8f6f7ae17e0fc8d11ea56d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE"></span><span id="tensorrt_llm::runtime::DecodingOutput::beamHypotheses__BeamHypotheses"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a1e1c48231792a45618deff974b48b4ee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"><span class="n"><span class="pre">BeamHypotheses</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamHypotheses</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE"></span><span id="tensorrt_llm::runtime::DecodingOutput::kNegativeInfinity__float"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1afad8e90f48ab5da093fdbb84b801f6da"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kNegativeInfinity</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1e20f</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamHypotheses</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::empty__BufferManagerR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a00c82796e448818892e94f8a064a74cd"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">empty</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::reshape__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a5ca4a3c3e34fe1f94224866821a2c352"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reshape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::release"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a6c10ab322d436f29ce99f0f0cb4e1e1a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">release</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::init__BufferManagerR.TokenIdType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1ab42b472a1d818532cd173ba69ad6097e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">init</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE" title="tensorrt_llm::runtime::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::slice__SizeType.SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a35faed30b9315b8e1890c64d77561ac1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"><span class="n"><span class="pre">BeamHypotheses</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIndex</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::outputIdsTgt__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1aaeebc4faabecf671bd786c5ff44e2364"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">outputIdsTgt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::sequenceLengthsTgt__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a42ed879ae2b5475b54aa7d3328b5357b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLengthsTgt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::cumLogProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1ad2aaa38891d139affc6713f4d1a6e207"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::normedScores__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1af97b849bf039b95020a3e9d7601b77f0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">normedScores</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a51b9823628bb1388d261bbd18b4e141e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::minNormedScores__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a545ce50c9d6ba29ebd84bbe1c7379648"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">minNormedScores</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::numBeams__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a207a33747d2474bc9252b6105e1cf7e9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numBeams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::isDone__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a75ad5911c7aa09ed3214d0d2ee2e4bad"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isDone</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="generationinput-h">
<h2>generationInput.h<a class="headerlink" href="#generationinput-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE">
<span id="_CPPv3I00EN12tensorrt_llm7runtime22GenericGenerationInputE"></span><span id="_CPPv2I00EN12tensorrt_llm7runtime22GenericGenerationInputE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TTensor</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningParams</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericGenerationInput</span></span></span><a class="headerlink" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a9697fefb8d30d140fb950c041cd3c86b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE" title="tensorrt_llm::runtime::GenericGenerationInput::TTensor"><span class="n"><span class="pre">TTensor</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput22GenericGenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput22GenericGenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput22GenericGenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::GenericGenerationInput__SizeTypeC.SizeTypeC.TensorPtr.TensorPtr.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a1fa8815906f92b622267f523a81597c0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericGenerationInput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">lengths</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">packed</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput22GenericGenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5endIdE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput5endIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput5endIdE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::endId__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a5623edc94ce2e11f4df7aad8a0c84ad9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5endIdE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5padIdE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput5padIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput5padIdE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::padId__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1adf1b22c075502a3a0556897cc1b6c91e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">padId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput5padIdE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput3idsE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a33a188aa66efd03ca14459152178d6dd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput3idsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput7lengthsE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a6e25b8ca6502844fa99acc5ed8cb4cf0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput7lengthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput6packedE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput6packedE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput6packedE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::packed__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a1251ffbef413840f81dc59bdedb88ee1"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">packed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput6packedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13embeddingBiasE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput13embeddingBiasE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput13embeddingBiasE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::embeddingBias__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a69f8b8593ce1a89fd78f907b4c17cf55"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBias</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13embeddingBiasE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput12badWordsListE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a7fdba88ec2c4ef3a749bf6730a8427f4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12badWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput13stopWordsListE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a13e840f74710c9ac836c64797652ebbe"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput13stopWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12maxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput12maxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput12maxNewTokensE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::maxNewTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1aa56894695f6cdf383655e1e01fe8c3a3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput12maxNewTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput18promptTuningParamsE">
<span id="_CPPv3N12tensorrt_llm7runtime22GenericGenerationInput18promptTuningParamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime22GenericGenerationInput18promptTuningParamsE"></span><span id="tensorrt_llm::runtime::GenericGenerationInput::promptTuningParams__PromptTuningParams"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationInput_1a4da27c7a8e7b4bdd90c87c4bf9a08b46"></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE" title="tensorrt_llm::runtime::GenericGenerationInput::PromptTuningParams"><span class="n"><span class="pre">PromptTuningParams</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">promptTuningParams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime22GenericGenerationInput18promptTuningParamsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInputE"></span><span id="tensorrt_llm::runtime::GenerationInput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationInput</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE" title="tensorrt_llm::runtime::GenericGenerationInput"><span class="n"><span class="pre">GenericGenerationInput</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParamsE" title="tensorrt_llm::runtime::PromptTuningParams"><span class="n"><span class="pre">PromptTuningParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput4BaseE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput4BaseE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput4BaseE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1ac4231ce739fe28958d51d834024d92cd"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Base</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime22GenericGenerationInputE" title="tensorrt_llm::runtime::GenericGenerationInput"><span class="n"><span class="pre">GenericGenerationInput</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParamsE" title="tensorrt_llm::runtime::PromptTuningParams"><span class="n"><span class="pre">PromptTuningParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput4BaseE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1ad525a655e0d2658c3f99dabb572acf1a"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput4BaseE" title="tensorrt_llm::runtime::GenerationInput::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">TensorPtr</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="tensorrt_llm::runtime::GenerationInput::GenerationInput__SizeTypeC.SizeTypeC.TensorPtr.TensorPtr.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1a919fd8de0e3905a059b746f1e9e35dbe"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationInput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">lengths</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">packed</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="generationoutput-h">
<h2>generationOutput.h<a class="headerlink" href="#generationoutput-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime23GenericGenerationOutputE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime23GenericGenerationOutputE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TTensor</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericGenerationOutput</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a739b5cbc24692f24b77a467612857e35"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE" title="tensorrt_llm::runtime::GenericGenerationOutput::TTensor"><span class="n"><span class="pre">TTensor</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a6baa6c81999b6ed0748db81b7c820c78"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Callback</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n"><span class="pre">ids</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n"><span class="pre">step</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n"><span class="pre">finished</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput23GenericGenerationOutputE9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput23GenericGenerationOutputE9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput23GenericGenerationOutputE9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::GenericGenerationOutput__TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a3ed16459b1913958c761070564be3754"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericGenerationOutput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">lengths</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput23GenericGenerationOutputE9TensorPtr9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput3idsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a7ff8e99b46a473dbfc53aecc127449bd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput3idsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput7lengthsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a53bb028de3dd943d7d0913753e522aae"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput7lengthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput11cumLogProbsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::cumLogProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a3f7b1e14ccdaa0c7525f90d32e1c43d6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput11cumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput8logProbsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1ae28cb59770ea150c435484b4f7f453d3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8logProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput13contextLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput13contextLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput13contextLogitsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::contextLogits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a884e4fabdeb57a635f796ddc36274abc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput13contextLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16generationLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput16generationLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput16generationLogitsE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::generationLogits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1ad76f135de78a3ab89804cbf3f7175d0f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenericGenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16generationLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16onTokenGeneratedE">
<span id="_CPPv3N12tensorrt_llm7runtime23GenericGenerationOutput16onTokenGeneratedE"></span><span id="_CPPv2N12tensorrt_llm7runtime23GenericGenerationOutput16onTokenGeneratedE"></span><span id="tensorrt_llm::runtime::GenericGenerationOutput::onTokenGenerated__Callback"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericGenerationOutput_1a35249ba2dc1990149ddc5895c2e3a1f7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput8CallbackE" title="tensorrt_llm::runtime::GenericGenerationOutput::Callback"><span class="n"><span class="pre">Callback</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">onTokenGenerated</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime23GenericGenerationOutput16onTokenGeneratedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutputE"></span><span id="tensorrt_llm::runtime::GenerationOutput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationOutput</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE" title="tensorrt_llm::runtime::GenericGenerationOutput"><span class="n"><span class="pre">GenericGenerationOutput</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput4BaseE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput4BaseE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput4BaseE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1ad51e9ff641b7853ab71d1aa3d584bc68"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Base</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime23GenericGenerationOutputE" title="tensorrt_llm::runtime::GenericGenerationOutput"><span class="n"><span class="pre">GenericGenerationOutput</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput4BaseE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a5c7e8fe9eade02e97e6576ecfe16b0df"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput4BaseE" title="tensorrt_llm::runtime::GenerationOutput::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">TensorPtr</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::GenerationOutput::GenerationOutput__TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a0c21ff4190e449fe91a596650c8d2146"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationOutput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">lengths</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptdecoder-h">
<h2>gptDecoder.h<a class="headerlink" href="#gptdecoder-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm6layersE">
<span id="_CPPv3N12tensorrt_llm6layersE"></span><span id="_CPPv2N12tensorrt_llm6layersE"></span><span id="tensorrt_llm::layers"></span><span class="target" id="namespacetensorrt__llm_1_1layers"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">layers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm6layersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoderE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoderE"></span><span id="tensorrt_llm::runtime::IGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="Link to this definition"></a><br /></dt>
<dd><p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1GptDecoder"><span class="std std-ref">tensorrt_llm::runtime::GptDecoder&lt; T &gt;</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoderD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoderD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoderD0Ev"></span><span id="tensorrt_llm::runtime::IGptDecoder::~IGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a5ad01b4d1a088b47e56ce6bed969cb57"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~IGptDecoder</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoder::setup__SamplingConfigCR.s.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1ab380759fa6e6e33a4a741d876e97298f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::IGptDecoder::forward__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a892c655936e25a38a7f0737fbfadfc1d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::IGptDecoder::forwardAsync__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a8f87707fea456a6decd013b6b831d336"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="tensorrt_llm::runtime::IGptDecoder::gatherTree__ITensorR.DecodingOutputCR.DecodingInputCR.BufferManagerCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a3b5ff6b13809e1049979ce441c249fca"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gatherTree</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finalOutputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingOutput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingInput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder17getSamplingConfigEv">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder17getSamplingConfigEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder17getSamplingConfigEv"></span><span id="tensorrt_llm::runtime::IGptDecoder::getSamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a15f6c7ed57d546c2bb0b1a50f3a6b9c7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder17getSamplingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder22acceptDraftTokensByIdsERK7ITensorRK7ITensorRK7ITensorRK7ITensorR7ITensorRK7ITensorR7ITensorR7ITensorRKN13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder22acceptDraftTokensByIdsERK7ITensorRK7ITensorRK7ITensorRK7ITensorR7ITensorRK7ITensorR7ITensorR7ITensorRKN13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder22acceptDraftTokensByIdsERK7ITensorRK7ITensorRK7ITensorRK7ITensorR7ITensorRK7ITensorR7ITensorR7ITensorRKN13BufferManager13CudaStreamPtrE"></span><span id="tensorrt_llm::runtime::IGptDecoder::acceptDraftTokensByIds__ITensorCR.ITensorCR.ITensorCR.ITensorCR.ITensorR.ITensorCR.ITensorR.ITensorR.BufferManager::CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a3be6cbd2637cebdae881fa260755acef"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">acceptDraftTokensByIds</span></span></span><span class="sig-paren">(</span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">targetTokenIds</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">draftTokenIds</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">contextLengths</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">numDraftTokens</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sequenceLengths</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finishedVec</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finishedFinal</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finishedSum</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder22acceptDraftTokensByIdsERK7ITensorRK7ITensorRK7ITensorRK7ITensorR7ITensorRK7ITensorR7ITensorR7ITensorRKN13BufferManager13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder25acceptDraftTokensByLogitsER7ITensorRK7ITensorR7ITensorR7ITensorRK7ITensorR7ITensor8SizeType8SizeTypebfP13curandState_tRKN13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder25acceptDraftTokensByLogitsER7ITensorRK7ITensorR7ITensorR7ITensorRK7ITensorR7ITensor8SizeType8SizeTypebfP13curandState_tRKN13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder25acceptDraftTokensByLogitsER7ITensorRK7ITensorR7ITensorR7ITensorRK7ITensorR7ITensor8SizeType8SizeTypebfP13curandState_tRKN13BufferManager13CudaStreamPtrE"></span><span id="tensorrt_llm::runtime::IGptDecoder::acceptDraftTokensByLogits__ITensorR.ITensorCR.ITensorR.ITensorR.ITensorCR.ITensorR.SizeType.SizeType.b.float.curandState_tP.BufferManager::CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1ac0e6eecf4416a6bdd1e176fedb0a86f1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">acceptDraftTokensByLogits</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">draftLogits</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">targetLogits</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">draftProbs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">targetProbs</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">numDraftTokens</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finished</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useRandomAcceptThreshold</span></span>, <span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">randomAcceptThreshold</span></span>, <span class="n"><span class="pre">curandState_t</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">curandState</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder25acceptDraftTokensByLogitsER7ITensorRK7ITensorR7ITensorR7ITensorRK7ITensorR7ITensor8SizeType8SizeTypebfP13curandState_tRKN13BufferManager13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE"></span><span id="tensorrt_llm::runtime::IGptDecoder::create__nvinfer1::DataType.s.s.BufferManager::CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1af9c668b9e5cdd2e0a65bfd55ba9b8b98"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">create</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10GptDecoderE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10GptDecoderE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoder</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a8b5c0f02247cc75b5d121681c1990d3b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ac3e1b92c75c404260a36d3d7e873231b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::GptDecoder::GptDecoder__s.s.CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a0cbdae7bd365d6175f9fe78a0237cc7c"></span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoder</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoder::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoder::setup__SamplingConfigCR.s.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a54668226efcc5825da42b2f21c80e7b4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::GptDecoder::forward__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a687dcb28ec2d720be9c35694250d346f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::GptDecoder::forwardAsync__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1af707cd75a441ad7c639536c75459c65e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="tensorrt_llm::runtime::GptDecoder::gatherTree__ITensorR.DecodingOutputCR.DecodingInputCR.BufferManagerCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ab5f5fba8712f080a59ae530dd8f9936b"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gatherTree</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finalOutputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingOutput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingInput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder17getSamplingConfigEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder17getSamplingConfigEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder17getSamplingConfigEv"></span><span id="tensorrt_llm::runtime::GptDecoder::getSamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a92d573f397311075a0c0f121bf774f60"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder17getSamplingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder8mManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder8mManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder8mManagerE"></span><span id="tensorrt_llm::runtime::GptDecoder::mManager__BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a8602244879ca7e9a83ccb7811cea26a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder8mManagerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder10mAllocatorE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder10mAllocatorE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder10mAllocatorE"></span><span id="tensorrt_llm::runtime::GptDecoder::mAllocator__common::CudaAllocator"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a51093c91db9709a5cdb1d6004c19ae57"></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CudaAllocator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mAllocator</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10mAllocatorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE"></span><span id="tensorrt_llm::runtime::GptDecoder::mDynamicDecodeLayer__std::shared_ptr:tensorrt_llm::layers::DynamicDecodeLayer:T::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ae47acd96842e4ebc4f34af4fc4370746"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm6layersE" title="tensorrt_llm::layers"><span class="n"><span class="pre">layers</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DynamicDecodeLayer</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE" title="tensorrt_llm::runtime::GptDecoder::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDynamicDecodeLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder14mLogProbsTiledE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder14mLogProbsTiledE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder14mLogProbsTiledE"></span><span id="tensorrt_llm::runtime::GptDecoder::mLogProbsTiled__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ac4a614d15044a39d9010ea86035ca326"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder9TensorPtrE" title="tensorrt_llm::runtime::GptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogProbsTiled</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder14mLogProbsTiledE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder15mSamplingConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder15mSamplingConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder15mSamplingConfigE"></span><span id="tensorrt_llm::runtime::GptDecoder::mSamplingConfig__SamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ac921712531dd8bb357fe6787f036db97"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSamplingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder15mSamplingConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptdecoderbatch-h">
<h2>gptDecoderBatch.h<a class="headerlink" href="#gptdecoderbatch-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatchE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatchE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatchE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderBatch</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE" title="tensorrt_llm::runtime::IGptDecoderBatch"><span class="n"><span class="pre">IGptDecoderBatch</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatchE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;gptDecoderBatch.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a8206454db9d639c251910c03681a9ed0"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a593596978ca3ce5ce4f500702a42182e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::GptDecoderBatch__std::s.std::s.CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1af478d2ea92161498aefa5ddb70d36262"></span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderBatch</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::setup__SizeType.SizeType.SizeType.SizeType.SizeType.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1af9c4853dc270c1ff0e802f72192ab0aa"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxKvCacheLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokensPerStep</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Setup the decoder before calling <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ab7e07276f33bfa6ef3c90baa49ee6eec"><span class="std std-ref"><span class="pre">forward()</span></span></a></code></p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::newRequest__SizeType.decoder_batch::RequestCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7214c20bc16c253fd2cb199884cd9859"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="tensorrt_llm::runtime::decoder_batch::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Initialize the decoder at <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code> with a new <code class="docutils literal notranslate"><span class="pre">request</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::newBatch__GenerationInputCR.GenerationOutputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a050019ac1971ebc1a8c816d9dd22d7b4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Initialize the decoder with new batch of inputs. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardAsync__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aa468001b519ad9767958cb8e660faea5"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">TokenPtr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host process and return the token for synchronization. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardSync__decoder_batch::TokenCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a495a39bf71a7d39cd81495834716760f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">e</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for the call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> associated with a token to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardAsync__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab0220317a75f65030f59878e26e3d737"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host thread. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncEv">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncEv"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardSync"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4cfcadb7da525a74e3fa3143ccd56b53"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncEv" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for the last call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4c69dab125603dd2a4fbd9ab043afa30"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize], indicators of finished requests </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a313a9ae3a464f84398b9ba038b4af554"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>batchIdx</strong> – index of the batch </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a445fc8f3f0db689b2166cb15fdc63df2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::finalize__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a95478c97710c43b25c91dbe3aa9544b3"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Gather final beam search results for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>. Result will only be available after event returned. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::finalizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a28208992ea3e90e60712d07ed05e8a72"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch8finalizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Gather final beam search results for all requests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getParentIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a93178ec5242faf2de970d4d017352dae"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParentIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains parent ids collected during beam search without padding, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getCumLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a3c07c5204d41a4dfdf688a4482d510bb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getCumLogProbs__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac93b3755a416298a93a7cc7263c49c9c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9c8cbfc561d34e459d677c1031588cf9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getLogProbs__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a92033458c848a4f94e7ab2c4f9516c1e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogProbs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getLogProbsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch15getAllNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch15getAllNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch15getAllNewTokensEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getAllNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a8217e98da5b107d12401590d80eeb976"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getAllNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch15getAllNewTokensEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get maxTokensPerStep tokens generated in the last forward pass. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxTokensPerStep, batchSize, maxBeamWidth], tokens generated in last forward pass, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNewTokens__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac08d8a77d78b71ff527ecd5999cc16d2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNewTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iter</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Get tokens generated in one step of last forward pass. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>iter</strong> – The iteration within [0; maxTokensPerStep) for which to get the tokens </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>[batchSize, beamWidth], tokens generated in <code class="docutils literal notranslate"><span class="pre">iter</span></code> (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNbStepsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9c9a6a07562637a290ed93e987bd06a7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbSteps</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize], the number of generation steps executed on each request </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNbFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7c4d967f7646e5a78f50d4ac6844be93"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[1], number of finished sequences, in pinned host memory </p>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1adbc3e2ab6fcff2a86fee1974fd2ea358"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aebed7ff7ea091a472908a7d2cc468684"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInputPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a8b297105bd88a4066694adf8a91d4073"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutputPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::postProcessRequest__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7c4def0f8b1239b53724fbf6cf576b12"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">postProcessRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Gather final beam search results for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mVocabSize__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a65caae45851a98064c90efb4e69fd5be"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mVocabSizePadded__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a6a3edc3073a3affe187f9344568d5349"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSizePadded</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mStream__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a33675102b6406b35993f06c59508b4fd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mBufferManager__BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a73a99bf969ac6a13872b1b93f443bc2d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBufferManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mForwardToken__TokenPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1abe39442ac571c5827d98ec295bd0513c"></span><span class="n"><span class="pre">TokenPtr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mForwardToken</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mForwardEvent__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9d42242aa2aac2bffd0387a6272f57f9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mForwardEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mStreams__std::vector:CudaStreamPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a5b1a9b00ef30809bef69336e5adfcc6c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStreams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecoders__std::vector:GptDecoderPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a656cea9abf1ea0c8b3697a79ab51dde9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::GptDecoderPtr"><span class="n"><span class="pre">GptDecoderPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoders</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecodingInputs__std::vector:DecodingInputPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aeb33339a3b75a2152134fdacd2660786"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingInputPtr"><span class="n"><span class="pre">DecodingInputPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingInputs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecodingOutputs__std::vector:DecodingOutputPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab1c5ab4afe168288cb33331a486d5be8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingOutputPtr"><span class="n"><span class="pre">DecodingOutputPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingOutputs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingInput__DecodingInputPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a73642bc20fd9bcd68b262edd55d4d2a7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingInputPtr"><span class="n"><span class="pre">DecodingInputPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mJointDecodingInput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingOutput__DecodingOutputPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a09890f57577749180eb23ba792989e96"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingOutputPtr"><span class="n"><span class="pre">DecodingOutputPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mJointDecodingOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mDraftTokenIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch14mDraftTokenIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch14mDraftTokenIdsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDraftTokenIds__std::vector:TensorPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aae2a553d3c59b1f3d0de3c1205a68673"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDraftTokenIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mDraftTokenIdsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mDraftLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12mDraftLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12mDraftLogitsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDraftLogits__std::vector:TensorPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a1d8cc482b234869894cb75802a84457d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDraftLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mDraftLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mAcceptByLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15mAcceptByLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15mAcceptByLogitsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mAcceptByLogits__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1add4954c40b66f5cd29f15326d83f0d23"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mAcceptByLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mAcceptByLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mNumDraftTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15mNumDraftTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15mNumDraftTokensE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mNumDraftTokens__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac9a6914fb43e482d6da7c26ce69f0519"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumDraftTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mNumDraftTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mCurandStatesE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mCurandStatesE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mCurandStatesE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mCurandStates__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a182eb5969447169037eadb734138e6b7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCurandStates</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mCurandStatesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mNbSteps__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab82d738a6ee4eb24160c4203f50ba47b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbSteps</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mFinished__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4bacfb74153463dd2ea29e9e58d62139"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFinished</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mFinishedSum__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a5e29dd7a16e6dcfdbc6af1f5a30d454d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFinishedSum</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxNewTokens__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1adda26d8626cb810bd54f0eae099f81d6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mBeamWidths__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a62af2790978fdbdce912bb55d1538c7a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamWidths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch23mGeneratedTokensPerStepE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch23mGeneratedTokensPerStepE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch23mGeneratedTokensPerStepE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mGeneratedTokensPerStep__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a88c38b63243a4f3b2f2593dc57803be2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGeneratedTokensPerStep</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch23mGeneratedTokensPerStepE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mFinishedStepsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch14mFinishedStepsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch14mFinishedStepsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mFinishedSteps__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1abe9d67818cff6708e78b0b23a010fe59"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFinishedSteps</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mFinishedStepsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mDraftProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11mDraftProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11mDraftProbsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDraftProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac91bf42fb15e82f745a929df086b9612"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDraftProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mDraftProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mTargetProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12mTargetProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12mTargetProbsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mTargetProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a545d6c68ca6c6dc7ea49dce14891630f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTargetProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mTargetProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxSequenceLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a17b9d45dd6c74bddca33b2edb555fe3c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxSequenceLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxKvCacheLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch17mMaxKvCacheLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch17mMaxKvCacheLengthE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxKvCacheLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab17a3d52e2c729cf7a2faf925efe78bf"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxKvCacheLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxKvCacheLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mActualBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7e2f18c7b2ebe3d76b568c46380e92c4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mActualBatchSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxTokensPerStepE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch17mMaxTokensPerStepE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch17mMaxTokensPerStepE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxTokensPerStep__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aab9825732279e411e112af18cb11a25e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxTokensPerStep</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17mMaxTokensPerStepE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptjsonconfig-h">
<h2>gptJsonConfig.h<a class="headerlink" href="#gptjsonconfig-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfigE"></span><span id="tensorrt_llm::runtime::GptJsonConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptJsonConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig"></span><span id="tensorrt_llm::runtime::GptJsonConfig::GptJsonConfig__ss.ss.SizeType.SizeType.GptModelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a66216254b0b7a635b772546973578fd8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptJsonConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">name</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a124fb0206f4ac3ff47629bcb8147589d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getModelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getNameC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a929bf268a1dd42fa06b6d05834bd6646"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getPrecisionC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ab5c3f3e5854ebdbd1b0f0d26fa8c2a6f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getPrecision</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getTensorParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1af3d5d313552e8497406ce3ae6b18bc11"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getPipelineParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a2ec48773fe7b568d771f53770d9b6b0d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getWorldSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a589461cb3bd53963212f1eda119d7f2e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWorldSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::engineFilename__WorldConfigCR.ssCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ac663861699ef7f573f64d5fdb89f14af"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">engineFilename</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">model</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig"></span><span id="tensorrt_llm::runtime::GptJsonConfig::engineFilename__WorldConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a9ce08952a809fbe5859a685215dd7258"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">engineFilename</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__ssCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a258c808e0e2c5445c807ae2f0a257782"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">json</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__isR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1af74ea4db9854b1f0ddd8aaa0d4a48f39"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">json</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__std::filesystem::pathCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a959c20c852124ced898b12bbdeb5d79f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">path</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5mNameE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5mNameE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5mNameE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mName__ssC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1aa23cc7f0c9ad465ffdb3c1950876b470"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mName</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5mNameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mPrecision__ssC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1acf01c9bba53cd18d0ece1026ef19e485"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPrecision</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mTensorParallelism__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a48b4d443ddb453b3349799486f020a3d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensorParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mPipelineParallelism__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ac469f1b66356d8ba919e63939aa26cf9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mGptModelConfig__GptModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a74a9d093ba911cfd9c09008a29124537"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGptModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptmodelconfig-h">
<h2>gptModelConfig.h<a class="headerlink" href="#gptmodelconfig-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfigE"></span><span id="tensorrt_llm::runtime::GptModelConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ModelVariant</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635aeb854c1219d60212d16ad06f0d454dc8"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGpt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635a30a41043cc5346b26f02b1a3e1cf54d5"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGlm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::GptModelConfig__SizeType.SizeType.SizeType.SizeType.nvinfer1::DataTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a93a4b8bc5df2e1a25d4b876fdefb1ff6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptModelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbLayers</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbHeads</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">hiddenSize</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getVocabSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aff992ef62ed1999326b48800913b2bee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVocabSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::getVocabSizePadded__SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1acf1d9e9818c254ee026c012388025480"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVocabSizePadded</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbLayers__SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a527a93c79413842c6706295232b0b9ed"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbLayers</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbHeadsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a00ef36e3eb9bd188a911b7f80c345e0f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbHeads</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbKvHeadsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a51a65e744ce4675cbdaaff83f16f2c54"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbKvHeads</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setNbKvHeads__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ad837f99f5aafdcc1ef7baa15c86617b5"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNbKvHeads</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbKvHeads</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getHiddenSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9df3d4f4da4ad6c30e6c91419d6b268f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHiddenSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getSizePerHeadCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a70409542e05a302a44ac2d571aea25db"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizePerHead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getDataTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab0dd6dd6a8b7cacfe4f60716066f8613"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::useGptAttentionPluginCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2eedc76ee7dc4712de143bcdc02f40f9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useGptAttentionPlugin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::useGptAttentionPlugin__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ac3723398406d29b02be8a06912ee0d82"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useGptAttentionPlugin</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useGptAttentionPlugin</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePackedInputCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a4894eb60ae226002fda29841977f7935"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePackedInput</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePackedInput__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1af71f4a0b8a8a8fc6e21ea28612b81ed1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePackedInput</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">inputPacked</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePagedKvCacheCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a173a85af1a71420f692612ca00c87d69"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePagedKvCache</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePagedKvCache__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a80165e232642897d7c4fb621bb61fed3"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePagedKvCache</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pagedKvCache</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getTokensPerBlockCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a0fa158a8e4b14102c6ee79a717d40879"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokensPerBlock</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setTokensPerBlock__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a443a5a8b94f03dd8b0a8fbbec298d3f8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTokensPerBlock</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">TokensPerBlock</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getQuantModeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa35c989f165bd8aa9437c3c6ba7d0a75"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getQuantMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::setQuantMode__common::QuantModeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a41fe2aa1fdd76ce76301859981425417"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setQuantMode</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">QuantMode</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::supportsInflightBatchingCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a275c4804ce2373f0c977c7a5f2ddd2a0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">supportsInflightBatching</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxBatchSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a24b66ade6d982663d81fa7b42d9c6055"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBatchSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxBatchSize__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab784bd1a00eb729f402abe8875b564ea"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBatchSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxInputLenCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1af1e370c6c5bbf9dfc854e12c14cccb79"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxInputLen</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxInputLen__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab80523799abed725a58348c4d56c8ad8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxInputLen</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxInputLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxOutputLenCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aaa48e78ace7bb49aa634f3cb84ed7ac4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxOutputLen</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxOutputLen__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab5e91bd0f16db6355ab0e0a14378658a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxOutputLen</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxOutputLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxNumTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5065aee21fa4e99f5c52bad69117a5cb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxNumTokens__std::optional:SizeType:CE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a06b325b04c1b396a0430320435f81373"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePromptTuningEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15usePromptTuningEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15usePromptTuningEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePromptTuningCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a14410c2939a2c4a7e82ec3d37cb5e767"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePromptTuning</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePromptTuningEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig30getMaxPromptEmbeddingTableSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig30getMaxPromptEmbeddingTableSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig30getMaxPromptEmbeddingTableSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxPromptEmbeddingTableSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a0e61bbb5092e714899e7296c10499ded"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxPromptEmbeddingTableSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig30getMaxPromptEmbeddingTableSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig30setMaxPromptEmbeddingTableSizeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig30setMaxPromptEmbeddingTableSizeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig30setMaxPromptEmbeddingTableSizeE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxPromptEmbeddingTableSize__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ae8c2698780f6ab43ccf08cf58730f395"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxPromptEmbeddingTableSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxPromptEmbeddingTableSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig30setMaxPromptEmbeddingTableSizeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeContextLogitsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2f0e4d2c822ec1a83c858674ffce5ec2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeContextLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeContextLogits__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a33188908109cfcc7baa824e284a1af8e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeContextLogits</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">computeContextLogits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeGenerationLogitsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ac92c3cb52036f92bfe70515fd828137b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeGenerationLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeGenerationLogits__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a715acca463b00c872f9d37b4dc0c1de9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeGenerationLogits</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">computeGenerationLogits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig23computeGenerationLogitsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getModelVariantC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9e832f202599100dfac6af143f8c72b4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getModelVariant</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant"></span><span id="tensorrt_llm::runtime::GptModelConfig::setModelVariant__ModelVariant"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5fd810a5a863b75656d6c0914b2d5762"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setModelVariant</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelVariant</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::useCustomAllReduceCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a392c5d57f071a1a36fa628580975faf9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useCustomAllReduce</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::useCustomAllReduce__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a79e7448c925f5d6785575250d7a1971f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useCustomAllReduce</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">customAllReduce</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxDraftLenE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14setMaxDraftLenE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14setMaxDraftLenE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxDraftLen__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ac96cb0cf174bb295ff5a2508d33ee749"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxDraftLen</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxDraftLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxDraftLenE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig19getMaxTokensPerStepEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig19getMaxTokensPerStepEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig19getMaxTokensPerStepEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxTokensPerStepCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ac202c324c51a94e74473a1ea2c264eac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxTokensPerStep</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig19getMaxTokensPerStepEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mVocabSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a18e1092a5c44a311577bc4954efc9ad8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbLayers__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a83dd559a542a2ebfe932387d4bc65faa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbLayers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbHeads__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a3aed4484b10e66d10c0f2e330aea1f06"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbHeads</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbKvHeads__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aafe290665c7ed003ec2347f04f1a54a5"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbKvHeads</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mHiddenSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a43e64ac4881c3a9eee58b7dee25964e2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHiddenSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mDataType__nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a497a3a345c3e48362790139c6fe56526"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mUseGptAttentionPlugin__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2c837cd8fccbcae0427ccb0334ddd9d2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUseGptAttentionPlugin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mInputPacked__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a780f8ca1c1f7565694104b0fa5bf45fb"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInputPacked</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mPagedKvCache__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2b70f372d6c64c92f1ad003043dcdb52"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPagedKvCache</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mTokensPerBlock__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa5ad1102f45507190c50ccfbd74f0788"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokensPerBlock</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mQuantMode__common::QuantMode"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9a5da6ac7818f7df4f68e3306653598b"></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mQuantMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a8714ccf704b0a20be40bb99e229c2439"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxInputLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab4be5cfe878d1333c6add3b824b6a7e4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxInputLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxOutputLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1adc7f72184ca681976df8197f475942c2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxOutputLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxNumTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa20bf7021d240c626fa6210511c31c47"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNumTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mComputeContextLogits__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a49736d7c35c69ae8eb50892e69dcfeb3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mComputeContextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig24mComputeGenerationLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig24mComputeGenerationLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig24mComputeGenerationLogitsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mComputeGenerationLogits__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a73f45ad3401a6c92d9b4727fb45b4485"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mComputeGenerationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig24mComputeGenerationLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mModelVariant__ModelVariant"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ae45c6e3f4a0ae12a5df7ba02281b544e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mModelVariant</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mUseCustomAllReduce__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2475ff83565a03603f3a3e61425320d6"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUseCustomAllReduce</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig28mMaxPromptEmbeddingTableSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig28mMaxPromptEmbeddingTableSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig28mMaxPromptEmbeddingTableSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxPromptEmbeddingTableSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a49acc82c17323e998fc0efdcc5e28ecd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxPromptEmbeddingTableSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig28mMaxPromptEmbeddingTableSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxDraftLenE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12mMaxDraftLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12mMaxDraftLenE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxDraftLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9e91f1f08560ab2853f0e983409daf45"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxDraftLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxDraftLenE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptsession-h">
<h2>gptSession.h<a class="headerlink" href="#gptsession-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_managerE"></span><span id="tensorrt_llm::batch_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">batch_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="tensorrt_llm::batch_manager::kv_cache_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager_1_1kv__cache__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kv_cache_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSessionE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSessionE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSessionE"></span><span id="tensorrt_llm::runtime::GptSession"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSessionE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9LoggerPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9LoggerPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1abf601de29903bb11af9d3d2976ea08a6"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoggerPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__ConfigCR.GptModelConfigCR.WorldConfigCR.voidCP.std::s.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a8f3776eb2f369f36460b77e660078686"></span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE" title="tensorrt_llm::runtime::GptSession::Config"><span class="n"><span class="pre">Config</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sessionConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">engineSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__ConfigCR.GptModelConfigCR.WorldConfigCR.std::vector:uint8_t:CR.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a08881cdd06f59f2f5988bd267aafcf99"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE" title="tensorrt_llm::runtime::GptSession::Config"><span class="n"><span class="pre">Config</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sessionConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__ConfigCR.GptModelConfigCR.WorldConfigCR.ssCR.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a968c21746698cf88e117bbd0fb9e81fe"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE" title="tensorrt_llm::runtime::GptSession::Config"><span class="n"><span class="pre">Config</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sessionConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineFile</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK6ConfigRK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession9getLoggerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession9getLoggerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession9getLoggerEv"></span><span id="tensorrt_llm::runtime::GptSession::getLoggerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3fa9a3cb9db8934ff06b1df13063b2f3"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getLogger</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getLoggerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv"></span><span id="tensorrt_llm::runtime::GptSession::getBufferManagerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a57a73dc8c994ce28207da1331377a08b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getBufferManager</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession14getModelConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession14getModelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession14getModelConfigEv"></span><span id="tensorrt_llm::runtime::GptSession::getModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aafeb2d6a1276ed948457924d1a54e744"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getModelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getModelConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv"></span><span id="tensorrt_llm::runtime::GptSession::getWorldConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a8921d43edbec6aa0b9ba6a6894505542"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getWorldConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession9getDeviceEv"></span><span id="tensorrt_llm::runtime::GptSession::getDeviceC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a94905fac580c8d0cd5b26b7fe319e853"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getDeviceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptSession::generate__GenerationOutputR.GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a55d90d8c6fccd7de2975d20ded7e4e42"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14KvCacheManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14KvCacheManagerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a17734c9adff9c0c3dd37d104c9c20879"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheManager</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="tensorrt_llm::batch_manager"><span class="n"><span class="pre">batch_manager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="tensorrt_llm::batch_manager::kv_cache_manager"><span class="n"><span class="pre">kv_cache_manager</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">KVCacheManager</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13KvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13KvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13KvCacheConfigE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a14e12e3a0354f7faa6497a29fb8a4c90"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheConfig</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="tensorrt_llm::batch_manager"><span class="n"><span class="pre">batch_manager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="tensorrt_llm::batch_manager::kv_cache_manager"><span class="n"><span class="pre">kv_cache_manager</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">KvCacheConfig</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13KvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3f4d6f00907ec557418c31614d991df4"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a15194437f183d58237ab0c8af3878b23"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenGeneratedCallback</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n"><span class="pre">step</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n"><span class="pre">finished</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13useCudaGraphsEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13useCudaGraphsEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13useCudaGraphsEv"></span><span id="tensorrt_llm::runtime::GptSession::useCudaGraphs"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1adae7d88300e62c0e18178dc9962ffd07"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useCudaGraphs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13useCudaGraphsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession15generateBatchedERNSt6vectorI16GenerationOutputEERKNSt6vectorI15GenerationInputEERK14SamplingConfigRK22TokenGeneratedCallback">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession15generateBatchedERNSt6vectorI16GenerationOutputEERKNSt6vectorI15GenerationInputEERK14SamplingConfigRK22TokenGeneratedCallback"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession15generateBatchedERNSt6vectorI16GenerationOutputEERKNSt6vectorI15GenerationInputEERK14SamplingConfigRK22TokenGeneratedCallback"></span><span id="tensorrt_llm::runtime::GptSession::generateBatched__std::vector:GenerationOutput:R.std::vector:GenerationInput:CR.SamplingConfigCR.TokenGeneratedCallbackCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a7fa116048db82b70aa2c733ba4fd16c8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generateBatched</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchesOutputs</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchesInputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE" title="tensorrt_llm::runtime::GptSession::TokenGeneratedCallback"><span class="n"><span class="pre">TokenGeneratedCallback</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">onTokenGenerated</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15generateBatchedERNSt6vectorI16GenerationOutputEERKNSt6vectorI15GenerationInputEERK14SamplingConfigRK22TokenGeneratedCallback" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession5setupERK6Config">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession5setupERK6Config"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession5setupERK6Config"></span><span id="tensorrt_llm::runtime::GptSession::setup__ConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a1dbeea870aaae467ecfe6961bb2bf259"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE" title="tensorrt_llm::runtime::GptSession::Config"><span class="n"><span class="pre">Config</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sessionConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession5setupERK6Config" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType8SizeTypeb">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType8SizeTypeb"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType8SizeTypeb"></span><span id="tensorrt_llm::runtime::GptSession::createContexts__SizeType.SizeType.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ae3759d6bf2b60f7ac4a7959431d278a5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createContexts</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numBatchesCtx</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numBatchesGen</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useCudaGraphs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType8SizeTypeb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createBuffers__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a9a0d4e7b740eff3848b6cb2d15d99608"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createBuffers</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createDecoders__SizeType.SizeType.SizeType.SizeType.nvinfer1::DataType.b.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a137882599d3b4929e862332bab1d6961"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createDecoders</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxKvCacheLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsType</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decoderPerRequest</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession20createKvCacheManagerE8SizeType8SizeType8SizeType8SizeTypeRK13KvCacheConfig">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession20createKvCacheManagerE8SizeType8SizeType8SizeType8SizeTypeRK13KvCacheConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession20createKvCacheManagerE8SizeType8SizeType8SizeType8SizeTypeRK13KvCacheConfig"></span><span id="tensorrt_llm::runtime::GptSession::createKvCacheManager__SizeType.SizeType.SizeType.SizeType.KvCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ab48c7974604dc50f4ea6a74c7c0a2899"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createKvCacheManager</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxKvCacheLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13KvCacheConfigE" title="tensorrt_llm::runtime::GptSession::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession20createKvCacheManagerE8SizeType8SizeType8SizeType8SizeTypeRK13KvCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createCustomAllReduceWorkspace__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a761e1bcaecfcd45d43b9737a97f88ef4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createCustomAllReduceWorkspace</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession18executeContextStepERKNSt6vectorI15GenerationInputEERKNSt6vectorI8SizeTypeEEPK14KvCacheManager">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession18executeContextStepERKNSt6vectorI15GenerationInputEERKNSt6vectorI8SizeTypeEEPK14KvCacheManager"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession18executeContextStepERKNSt6vectorI15GenerationInputEERKNSt6vectorI8SizeTypeEEPK14KvCacheManager"></span><span id="tensorrt_llm::runtime::GptSession::executeContextStep__std::vector:GenerationInput:CR.std::vector:SizeType:CR.KvCacheManagerCP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ab9c9713ffbacf77261cc0a71d9b04414"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executeContextStep</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatches</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchOffsets</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="tensorrt_llm::runtime::GptSession::KvCacheManager"><span class="n"><span class="pre">KvCacheManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">kvCacheManager</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession18executeContextStepERKNSt6vectorI15GenerationInputEERKNSt6vectorI8SizeTypeEEPK14KvCacheManager" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession21executeGenerationStepE8SizeTypeRKNSt6vectorI15GenerationInputEERNSt6vectorI16GenerationOutputEERKNSt6vectorI8SizeTypeEEP14KvCacheManagerRNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession21executeGenerationStepE8SizeTypeRKNSt6vectorI15GenerationInputEERNSt6vectorI16GenerationOutputEERKNSt6vectorI8SizeTypeEEP14KvCacheManagerRNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession21executeGenerationStepE8SizeTypeRKNSt6vectorI15GenerationInputEERNSt6vectorI16GenerationOutputEERKNSt6vectorI8SizeTypeEEP14KvCacheManagerRNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::GptSession::executeGenerationStep__SizeType.std::vector:GenerationInput:CR.std::vector:GenerationOutput:R.std::vector:SizeType:CR.KvCacheManagerP.std::vector:b:R"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aaca25dd333bb7b4f2c4f5ddf700edacd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executeGenerationStep</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">step</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchesInputs</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchesOutputs</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchOffsets</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="tensorrt_llm::runtime::GptSession::KvCacheManager"><span class="n"><span class="pre">KvCacheManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">kvCacheManager</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">microBatchesFinished</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession21executeGenerationStepE8SizeTypeRKNSt6vectorI15GenerationInputEERNSt6vectorI16GenerationOutputEERKNSt6vectorI8SizeTypeEEP14KvCacheManagerRNSt6vectorIbEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16decoderStepAsyncE8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16decoderStepAsyncE8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16decoderStepAsyncE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::decoderStepAsync__SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ab03cc4c728c2da8c0767efdc515af402"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoderStepAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">decoderStep</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16decoderStepAsyncE8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Execute decoder on last PP rank, receive decoder output on other PP ranks. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::shouldStopSync__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a4d2ef8994105f56856cd3e44666611e2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shouldStopSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Synchronize with the decoder and return the <code class="docutils literal notranslate"><span class="pre">shouldStop</span></code> flag. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8finalizeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8finalizeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8finalizeE8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::finalize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a49b547cc837f1c9dde99201e90a71156"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8finalizeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Collect final output ids and log probs on last PP rank and send them to first PP rank. </p>
<p>Receives are asynchronous on host, so synchronization is required before access. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::kvCacheAddSequences__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a35035e8a6a38dd38f94571fb237721c8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheAddSequences</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">firstBatchIdx</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession11initDecoderER7ITensorRK15GenerationInputRK16GenerationOutputRK14SamplingConfig8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession11initDecoderER7ITensorRK15GenerationInputRK16GenerationOutputRK14SamplingConfig8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession11initDecoderER7ITensorRK15GenerationInputRK16GenerationOutputRK14SamplingConfig8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::initDecoder__ITensorR.GenerationInputCR.GenerationOutputCR.SamplingConfigCR.SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a25e761e649ffe025d0df642d71544501"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">initDecoder</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession11initDecoderER7ITensorRK15GenerationInputRK16GenerationOutputRK14SamplingConfig8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Populate outputIds and return reference to newTokens tensor. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession30createOnTokenGeneratedCallbackER16GenerationOutput">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession30createOnTokenGeneratedCallbackER16GenerationOutput"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession30createOnTokenGeneratedCallbackER16GenerationOutput"></span><span id="tensorrt_llm::runtime::GptSession::createOnTokenGeneratedCallback__GenerationOutputR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ac7fab944335f448ad3296ebe39f9ef3f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession22TokenGeneratedCallbackE" title="tensorrt_llm::runtime::GptSession::TokenGeneratedCallback"><span class="n"><span class="pre">TokenGeneratedCallback</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createOnTokenGeneratedCallback</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession30createOnTokenGeneratedCallbackER16GenerationOutput" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession12mModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession12mModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession12mModelConfigE"></span><span id="tensorrt_llm::runtime::GptSession::mModelConfig__GptModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1acd20c0377c38abb8707d0fcc7b3c1b97"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mModelConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession12mWorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession12mWorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession12mWorldConfigE"></span><span id="tensorrt_llm::runtime::GptSession::mWorldConfig__WorldConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1af712d3cd32f2c0952c6cb5bf1cfece45"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mWorldConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession7mDeviceE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession7mDeviceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession7mDeviceE"></span><span id="tensorrt_llm::runtime::GptSession::mDevice__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a557d94b8bbfbdc168c417eeae307c25a"></span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDevice</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mDeviceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13mPipelineCommE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13mPipelineCommE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13mPipelineCommE"></span><span id="tensorrt_llm::runtime::GptSession::mPipelineComm__std::shared_ptr:NcclCommunicator:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1afd92278f3aa9e57d02c58f5c5cb1de3a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">NcclCommunicator</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineComm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13mPipelineCommE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession11mCommStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession11mCommStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession11mCommStreamE"></span><span id="tensorrt_llm::runtime::GptSession::mCommStream__std::shared_ptr:CudaStream:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a14480c0db4c1615ed788836285a3c1e7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession11mCommStreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10mCommEventE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10mCommEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10mCommEventE"></span><span id="tensorrt_llm::runtime::GptSession::mCommEvent__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a0916e45f7ddd9ac0a27eec4897260802"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommEvent</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10mCommEventE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9mCommPtrsE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9mCommPtrsE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9mCommPtrsE"></span><span id="tensorrt_llm::runtime::GptSession::mCommPtrs__ITensor::SharedPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a62395eff1a3bce743fc6906a95d5101c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommPtrs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9mCommPtrsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17mIpcMemoryHandlesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17mIpcMemoryHandlesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17mIpcMemoryHandlesE"></span><span id="tensorrt_llm::runtime::GptSession::mIpcMemoryHandles__std::vector:std::shared_ptr:IpcMemory::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a135834a47cb19851d6f8872b3a54c31a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryE" title="tensorrt_llm::runtime::IpcMemory"><span class="n"><span class="pre">IpcMemory</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mIpcMemoryHandles</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17mIpcMemoryHandlesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE"></span><span id="tensorrt_llm::runtime::GptSession::mDecoderMaxSequenceLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3eb70f53bb0cbc5c6e032c37520b101c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoderMaxSequenceLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession24mDecoderMaxKvCacheLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession24mDecoderMaxKvCacheLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession24mDecoderMaxKvCacheLengthE"></span><span id="tensorrt_llm::runtime::GptSession::mDecoderMaxKvCacheLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a88f99e921cbb753b535f814bb7ab9a9f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoderMaxKvCacheLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession24mDecoderMaxKvCacheLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession7mLoggerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession7mLoggerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession7mLoggerE"></span><span id="tensorrt_llm::runtime::GptSession::mLogger__LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a4bd2d4d4f201bf17a87ed40c4e29ce55"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogger</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mLoggerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8mRuntimeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8mRuntimeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8mRuntimeE"></span><span id="tensorrt_llm::runtime::GptSession::mRuntime__std::shared_ptr:TllmRuntime:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aa97fc0a02f3f28e3e2121e2e9bc10c56"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">TllmRuntime</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRuntime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mRuntimeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession15mKvCacheManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession15mKvCacheManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession15mKvCacheManagerE"></span><span id="tensorrt_llm::runtime::GptSession::mKvCacheManager__std::shared_ptr:KvCacheManager:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aeaf492a405a12cd82da00036d75838d0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="tensorrt_llm::runtime::GptSession::KvCacheManager"><span class="n"><span class="pre">KvCacheManager</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mKvCacheManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15mKvCacheManagerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17mMicroBatchConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17mMicroBatchConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17mMicroBatchConfigE"></span><span id="tensorrt_llm::runtime::GptSession::mMicroBatchConfig__MicroBatchConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a70891459edc318254e61e107ec5edcb2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE" title="tensorrt_llm::runtime::GptSession::MicroBatchConfig"><span class="n"><span class="pre">MicroBatchConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMicroBatchConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17mMicroBatchConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9mDecodersE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9mDecodersE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9mDecodersE"></span><span id="tensorrt_llm::runtime::GptSession::mDecoders__std::vector:std::shared_ptr:IStatefulGptDecoder::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ab3287006aede8e62f11db44d5f8997e6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="tensorrt_llm::runtime::IStatefulGptDecoder"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoders</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9mDecodersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8mBuffersE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8mBuffersE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8mBuffersE"></span><span id="tensorrt_llm::runtime::GptSession::mBuffers__std::vector:std::shared_ptr:RuntimeBuffers::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a96b0baf2000c1616dcdcecf836cdd143"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">RuntimeBuffers</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBuffers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mBuffersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession15mReceivedEventsE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession15mReceivedEventsE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession15mReceivedEventsE"></span><span id="tensorrt_llm::runtime::GptSession::mReceivedEvents__std::vector:CudaEvent:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a9253a1bccec0b1ab57bab7b7d5b6f5bb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mReceivedEvents</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15mReceivedEventsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE"></span><span id="tensorrt_llm::runtime::GptSession::mCudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a1bf104808d61798aeba1c87beeb2c101"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphMode</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE"></span><span id="tensorrt_llm::runtime::GptSession::mCudaGraphInstances__std::vector:CudaGraphExecutor:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1abf794ab36d1086fbf43a6a33db6e5181"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE" title="tensorrt_llm::runtime::GptSession::CudaGraphExecutor"><span class="n"><span class="pre">CudaGraphExecutor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphInstances</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">batch_manager::TrtGptModelV1</span></dt>
</dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6ConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6ConfigE"></span><span id="tensorrt_llm::runtime::GptSession::Config"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Config</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6ConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;gptSession.h&gt;</em></div>
<p>Configuration for session execution and buffer sizes. <code class="docutils literal notranslate"><span class="pre">generate</span></code> may be called with batch size and beam width smaller than the configured parameters. </p>
<p><code class="docutils literal notranslate"><span class="pre">maxBatchSize</span></code> will be divided by the number of micro batches to initialize each batch buffer. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config6ConfigE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config6ConfigE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config6ConfigE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::Config::Config__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a18da0794db4fcd78d86fcd251a271315"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Config</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config6ConfigE8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config12maxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config12maxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config12maxBatchSizeE"></span><span id="tensorrt_llm::runtime::GptSession::Config::maxBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a287bf1db2d227b17409414054fbfd72e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config12maxBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config12maxBeamWidthE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config12maxBeamWidthE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config12maxBeamWidthE"></span><span id="tensorrt_llm::runtime::GptSession::Config::maxBeamWidth__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1aaee696b7af4da45b54ab762b4eba2059"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxBeamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config12maxBeamWidthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config17maxSequenceLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config17maxSequenceLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config17maxSequenceLengthE"></span><span id="tensorrt_llm::runtime::GptSession::Config::maxSequenceLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a741c59327475ef8172e8fb68cb1b5260"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxSequenceLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config17maxSequenceLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config17decoderPerRequestE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config17decoderPerRequestE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config17decoderPerRequestE"></span><span id="tensorrt_llm::runtime::GptSession::Config::decoderPerRequest__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a3a5fc9eed846017ce119c4d624defab4"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoderPerRequest</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config17decoderPerRequestE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config13cudaGraphModeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config13cudaGraphModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config13cudaGraphModeE"></span><span id="tensorrt_llm::runtime::GptSession::Config::cudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a2584bcfc17050c7ce1370868ccb1e9b9"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cudaGraphMode</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config13cudaGraphModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config13kvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config13kvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config13kvCacheConfigE"></span><span id="tensorrt_llm::runtime::GptSession::Config::kvCacheConfig__KvCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1ab6b5dafa19a5fa0c82d979e17d0a9b0f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13KvCacheConfigE" title="tensorrt_llm::runtime::GptSession::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheConfig</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config13kvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config17ctxMicroBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config17ctxMicroBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config17ctxMicroBatchSizeE"></span><span id="tensorrt_llm::runtime::GptSession::Config::ctxMicroBatchSize__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a07a837ef0fe1417e9859470895f95988"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ctxMicroBatchSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config17ctxMicroBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession6Config17genMicroBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession6Config17genMicroBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession6Config17genMicroBatchSizeE"></span><span id="tensorrt_llm::runtime::GptSession::Config::genMicroBatchSize__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1Config_1a0843b7e8eaeec76fab3f19863502aec2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">genMicroBatchSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession6Config17genMicroBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaGraphExecutor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a0f76e8397ca3097bc3451e2251c4d6bd"></span><span class="sig-name descname"><span class="n"><span class="pre">CudaGraphExecutor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::~CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a0d609c019d81064a9cdf235352d7c25f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~CudaGraphExecutor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::hasInstance"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a11ce3f651b7e0c1dbba82460a85cfd3b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">hasInstance</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::clear"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a6bd0bf9ff37bb89cb06d03fb19b2de59"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">clear</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::prepareNextGraph__TllmRuntimeCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1ae8bcb9ec8982af90576b5fc24b6f5599"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">prepareNextGraph</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">TllmRuntime</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">runtime</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nextContextId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::launch__CudaStreamCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a9d32b4b5046b27364c3e265b433b62a4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">launch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::create__cudaGraph_tCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a368e5d334bee7c4afd9085fdbe98f02e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">create</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaGraph_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">graph</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::update__cudaGraph_tCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a776709d610ef7122a55d1a9dd211acb9"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">update</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaGraph_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">graph</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::uploadToStream__CudaStreamCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1af2cd5b05bdcf087f91696bc83daa90e0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">uploadToStream</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::mInstance__cudaGraphExec_t"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1ac8e98428b96096b5509dc101b66be172"></span><span class="n"><span class="pre">cudaGraphExec_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInstance</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MicroBatchConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigEv"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::MicroBatchConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a1eac3f1b519be9f762d23157d6b04444"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MicroBatchConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigE8SizeType8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigE8SizeType8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigE8SizeType8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::MicroBatchConfig__SizeType.SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a55ab0fe231a604121d4801890ac22b7e"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MicroBatchConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">genMicroBatchSize</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ctxMicroBatchSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig16MicroBatchConfigE8SizeType8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig12numCtxPerGenEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig12numCtxPerGenEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig12numCtxPerGenEv"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::numCtxPerGenCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a36ce805a188b296f338f411333bf9158"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxPerGen</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig12numCtxPerGenEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getCtxContextIdE8SizeType8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getCtxContextIdE8SizeType8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getCtxContextIdE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::getCtxContextId__SizeType.SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a9079c00bbea192f469364110544be9a8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCtxContextId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">generationBatchId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">contextBatchId</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getCtxContextIdE8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>First 2 * numGenBatches contexts are for generation phase, next numCtxBatches are for context phase. Use numCtxPerGen() contexts for the context batches of each generation batch. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getGenContextIdE8SizeType8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getGenContextIdE8SizeType8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getGenContextIdE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::getGenContextId__SizeType.SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a56e4fdda5114c792aaf96ce251bb4103"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGenContextId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">flipFlopId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">generationBatchId</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16MicroBatchConfig15getGenContextIdE8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>First 2 * numGenBatches contexts are for generation phase, flip-flop between 2 of them for each generation batch. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numCtxBatchesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numCtxBatchesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numCtxBatchesE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::numCtxBatches__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1aaf5c9ff509b390c78cabf6dbeb2c32b0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxBatches</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numCtxBatchesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numGenBatchesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numGenBatchesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numGenBatchesE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::numGenBatches__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a1519020a4d32b0dfd1770376b244e250"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGenBatches</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig13numGenBatchesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12ctxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12ctxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12ctxBatchSizeE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::ctxBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1abe899259672c5a9d1503af2a0356bbaa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ctxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12ctxBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12genBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12genBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12genBatchSizeE"></span><span id="tensorrt_llm::runtime::GptSession::MicroBatchConfig::genBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1MicroBatchConfig_1a012551c5c3785c45f4be19612f0e4a02"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">genBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16MicroBatchConfig12genBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime5utilsE">
<span id="_CPPv3N12tensorrt_llm7runtime5utilsE"></span><span id="_CPPv2N12tensorrt_llm7runtime5utilsE"></span><span id="tensorrt_llm::runtime::utils"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1utils"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">utils</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime5utilsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE"></span><span id="tensorrt_llm::runtime::utils::loadEngine__ssCR"></span><span class="target" id="gptSession_8h_1ae397b7fd72675fbc97c0e662cc71b5eb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">loadEngine</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">enginePath</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="ibuffer-h">
<h2>iBuffer.h<a class="headerlink" href="#ibuffer-h" title="Link to this heading"></a></h2>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE" title="tensorrt_llm::runtime::MemoryType::kGPU"><span class="n"><span class="pre">kGPU</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kGPU:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4_1adc272a863661fdf8ea08ad956ebe5975"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;GPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE" title="tensorrt_llm::runtime::MemoryType::kCPU"><span class="n"><span class="pre">kCPU</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kCPU:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4_1ad18d7d70b64c48821a5928c3eb1b6d0e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;CPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE" title="tensorrt_llm::runtime::MemoryType::kPINNED"><span class="n"><span class="pre">kPINNED</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kPINNED:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4_1a661ffce25b1fb3d0f97348ef5e92e6b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;PINNED&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kFLOAT</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1af6ef0c2bf073670006e1a68fd861495b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kFLOAT:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aff9156cc400ebaa290f874998c6ddc24"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;float&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kFLOAT:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aba2b5525ce1f152f1aaf69380203c672"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kFLOAT&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kFLOATEE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kHALF</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1a08c9427d16066ff8ee9e9deb35a91871"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">half</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kHALF:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1a5dcc983372abc63f1e73e0ef4efe3af4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;half&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kHALF:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1ae11a9f8c46eeece89d904c7ca51be48e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kHALF&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kHALFEE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1a51b8ee14d86b4b2a6cc5e57a1d524c43"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT8:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1aaf33f5d7fe9708fd4e85ed2d4a22c043"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int8&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT8:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1ad9bb9e89d790ebde4f26846c795f167a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kINT8&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kINT8EE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a37fa1b65a6bece3eb638f7693132053c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT32:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a51d6f667c98dc58d42c9b885299c6375"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int32&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT32:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1aad0e5753b4c85a9d8103f715aca31fa7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kINT32&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a288b116cbff3532cf2b80e7206277846"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT64:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a8ebfbe17f94412a4eed0de836c4cd55a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int64&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT64:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1ad8e7069321866172865f3464991ced3b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kINT64&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1a41a0f3bcdb82b843a9594fbe518c148d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXtrueEE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT32.true:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1aa736c483d9d0e4aa71d03e8c777e341c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint32&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXtrueEE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT32.true:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1a39eaff2ce1df3547b2790e42c61167f2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kINT32, true&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT32EXL1EEE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a8376cfb6029bfec1e7aaa9c29369b094"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXtrueEE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT64.true:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a869291a56676cdac6140f9846e99ffbf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint64&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXtrueEE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kINT64.true:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a50499242c7be43b3775a86737243d7c9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kINT64, true&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kINT64EXL1EEE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE">
<span id="_CPPv3I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE"></span><span id="_CPPv2I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kBOOL</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kBOOL, kUnsigned&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1a3ef7ec1c22f6cc0b397ab5eb28c232eb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kBOOL.kUnsigned:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1aba47b1869e577d85a3d93cc68b2ab24d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;bool&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kBOOL.kUnsigned:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1a1d0ef1feb1bbf08798efcdaaf934fcc0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kBOOL, kUnsigned&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType5kBOOLE9kUnsignedE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE">
<span id="_CPPv3I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE"></span><span id="_CPPv2I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kUINT8, kUnsigned&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_bEN12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1a5a3089fdad68ca4ce8bf36137d18a147"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kUINT8.kUnsigned:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1a31c93a3cf26ca1fbce36b588c146996c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint8&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:nvinfer1::DataType::kUINT8.kUnsigned:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1ad0a569389e9dda69079149e8ab016ab0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;nvinfer1::DataType::kUINT8, kUnsigned&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsIN8nvinfer18DataType6kUINT8E9kUnsignedE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4_1a996ff7940394a814ae164728312bf5ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4_1a10403a956a1d979e0cd10c8034e9f3e1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4_1a96b97cf0edb4ad984dd69278acfabd10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4_1af2839adf78453826b3a7f8422c9130fb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4_1a6d279398fbfedbbfffd0f7fbd1eb80b8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4_1a60d418bc6cea9f845ee353d78f0c3d66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime18PointerElementTypeE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime18PointerElementTypeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a6b78d5482ec51a801a9fe54db6eaa0f9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PointerElementType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_reference_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">element_type</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-enums">Enums</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryTypeE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryType</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType4kGPUE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType4kGPUE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869a37ae524b76587efa776affdc5cdf2ac1"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType4kCPUE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType4kCPUE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869a2ce310327f474afc9f6774faa2f57903"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType7kPINNEDE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType7kPINNEDE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869ae3278bcaa387e6baeef9b80c1e61c35a"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kPINNED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime16constPointerCastERKNSt10shared_ptrI1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime16constPointerCastERKNSt10shared_ptrI1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1ad7234909a7d00791f3d22a797078e9de"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_const_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">constPointerCast</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE">
<span id="_CPPv3I00EN12tensorrt_llm7runtime16constPointerCastERRNSt10unique_ptrI1T1DEE"></span><span id="_CPPv2I00EN12tensorrt_llm7runtime16constPointerCastERRNSt10unique_ptrI1T1DEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">D</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a1398d57cec6aacd5be041575ccb140fb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_const_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">constPointerCast</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::D"><span class="n"><span class="pre">D</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10bufferCastERK7IBuffer"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10bufferCastERK7IBuffer"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1aa39a24d2db1c10a205c75aa2c6a8f069"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer" title="tensorrt_llm::runtime::bufferCast::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">bufferCast</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10bufferCastER7IBuffer"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10bufferCastER7IBuffer"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a2f29317ef105d176296ef6b5518bcddf"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer" title="tensorrt_llm::runtime::bufferCast::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">bufferCast</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.IBufferCR"></span><span class="target" id="iBuffer_8h_1a4363785cd968e4e0cd1933cba37850b9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to print a buffer. </p>
</dd></dl>
</div>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kGPU</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError5valueE"></span><span id="tensorrt_llm::runtime::PhonyNameDueToError::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4_1adc272a863661fdf8ea08ad956ebe5975"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;GPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kCPU</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4_1ad18d7d70b64c48821a5928c3eb1b6d0e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;CPU&quot;</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kPINNED</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4_1a661ffce25b1fb3d0f97348ef5e92e6b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;PINNED&quot;</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime14DataTypeTraitsE">
<span id="_CPPv3I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime14DataTypeTraitsE"></span><span id="_CPPv2I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime14DataTypeTraitsE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDataType</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kIsUnsigned</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kIsPointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><a class="headerlink" href="#_CPPv4I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime14DataTypeTraitsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>For converting a TensorRT data type to a C++ data type. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kFLOAT</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1af6ef0c2bf073670006e1a68fd861495b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError4nameE"></span><span id="tensorrt_llm::runtime::PhonyNameDueToError::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aff9156cc400ebaa290f874998c6ddc24"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;float&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError4sizeE"></span><span id="tensorrt_llm::runtime::PhonyNameDueToError::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aba2b5525ce1f152f1aaf69380203c672"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kHALF</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1a08c9427d16066ff8ee9e9deb35a91871"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">half</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1a5dcc983372abc63f1e73e0ef4efe3af4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;half&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1ae11a9f8c46eeece89d904c7ca51be48e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT8</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1a51b8ee14d86b4b2a6cc5e57a1d524c43"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1aaf33f5d7fe9708fd4e85ed2d4a22c043"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int8&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1ad9bb9e89d790ebde4f26846c795f167a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT32</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a37fa1b65a6bece3eb638f7693132053c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a51d6f667c98dc58d42c9b885299c6375"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int32&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1aad0e5753b4c85a9d8103f715aca31fa7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT64</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a288b116cbff3532cf2b80e7206277846"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a8ebfbe17f94412a4eed0de836c4cd55a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;int64&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1ad8e7069321866172865f3464991ced3b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT32,</span> <span class="pre">true</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1a41a0f3bcdb82b843a9594fbe518c148d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1aa736c483d9d0e4aa71d03e8c777e341c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint32&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1a39eaff2ce1df3547b2790e42c61167f2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT64,</span> <span class="pre">true</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a8376cfb6029bfec1e7aaa9c29369b094"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a869291a56676cdac6140f9846e99ffbf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint64&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a50499242c7be43b3775a86737243d7c9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;bool</span> <span class="pre">kUnsigned&gt;</span> <span class="pre">kBOOL,</span> <span class="pre">kUnsigned</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1a3ef7ec1c22f6cc0b397ab5eb28c232eb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1aba47b1869e577d85a3d93cc68b2ab24d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;bool&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1a1d0ef1feb1bbf08798efcdaaf934fcc0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;bool</span> <span class="pre">kUnsigned&gt;</span> <span class="pre">kUINT8,</span> <span class="pre">kUnsigned</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1a5a3089fdad68ca4ce8bf36137d18a147"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1a31c93a3cf26ca1fbce36b588c146996c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;uint8&quot;</span></span><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1ad0a569389e9dda69079149e8ab016ab0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="tensorrt_llm::runtime::PhonyNameDueToError::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE">
<span id="_CPPv3I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE"></span><span id="_CPPv2I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDataType</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01kDataType_00_01kUnsigned_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataTypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;kDataType, kUnsigned, true&gt;::kDataType"><span class="n"><span class="pre">kDataType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;kDataType, kUnsigned, true&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01kDataType_00_01kUnsigned_00_01true_01_4_1a4e4f2844614eff1320acd710bc0cfe9c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">DataTypeTraits</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;kDataType, kUnsigned, true&gt;::kDataType"><span class="n"><span class="pre">kDataType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::DataTypeTraits&lt;kDataType, kUnsigned, true&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">type</span></span><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4nameE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4nameE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXtrueEE4nameE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:kDataType.kUnsigned.true:::name__cA"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01kDataType_00_01kUnsigned_00_01true_01_4_1a82b47cb338e9493046c041d4e511564d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">name</span></span></span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;*&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4nameE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4sizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4sizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXtrueEE4sizeE"></span><span id="tensorrt_llm::runtime::DataTypeTraits:kDataType.kUnsigned.true:::size__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1DataTypeTraits_3_01kDataType_00_01kUnsigned_00_01true_01_4_1a2c1daab28b706ec5735fb3b5367a733e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4typeE" title="tensorrt_llm::runtime::DataTypeTraits&lt;kDataType, kUnsigned, true&gt;::type"><span class="n"><span class="pre">type</span></span></a><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DataTypeTraitsI9kDataType9kUnsignedXL1EEE4sizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>A wrapper around <code class="docutils literal notranslate"><span class="pre">nvinfer1::DataType</span></code> that provides a support for pointer types. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb"></span><span id="tensorrt_llm::runtime::BufferDataType::BufferDataType__nvinfer1::DataType.b.bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a3cdb7e5d96b677f670d211cd3f0d2029"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferDataType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">_unsigned</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pointer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv"></span><span id="tensorrt_llm::runtime::BufferDataType::castto-nvinfer1::DataType-operatorCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1ae0e234f2e49d57ea1ec39bf76ef5f843"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv"></span><span id="tensorrt_llm::runtime::BufferDataType::getDataTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a375ef9858082b42a89a8fbddeb758836"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType9isPointerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType9isPointerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType9isPointerEv"></span><span id="tensorrt_llm::runtime::BufferDataType::isPointerCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a4f456474dbc06fcfb4c5fdd4f30c681b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isPointer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType9isPointerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv"></span><span id="tensorrt_llm::runtime::BufferDataType::isUnsignedCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a84da512a92aa9b9ba7498d8eface36d1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUnsigned</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType7getSizeEv"></span><span id="tensorrt_llm::runtime::BufferDataType::getSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a0e1c8d7eb345611735b5c8998efc1b29"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType7getSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType::kTrtPointerType__auto"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a10497366a0660dd533bd6bac96a65e97"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTrtPointerType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType9mDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType9mDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType9mDataTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType::mDataType__nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a57916add2426171d7066d14df4e796d8"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mDataTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType9mUnsignedE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType9mUnsignedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType9mUnsignedE"></span><span id="tensorrt_llm::runtime::BufferDataType::mUnsigned__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a675fe3124bb2de4dd047ce69a1a5a88b"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUnsigned</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mUnsignedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType8mPointerE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType8mPointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType8mPointerE"></span><span id="tensorrt_llm::runtime::BufferDataType::mPointer__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a494f3b4c8e08ef43209cac0e2f114e0c"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPointer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType8mPointerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0_bEN12tensorrt_llm7runtime11TRTDataTypeE">
<span id="_CPPv3I0_bEN12tensorrt_llm7runtime11TRTDataTypeE"></span><span id="_CPPv2I0_bEN12tensorrt_llm7runtime11TRTDataTypeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><a class="headerlink" href="#_CPPv4I0_bEN12tensorrt_llm7runtime11TRTDataTypeE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>For converting a C++ data type to a TensorRT data type. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIfEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIfEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01float_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:float:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01float_01_4_1a0a72c60c5a9b946e64a960d1075a5af9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kFLOAT</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01half_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">half</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:half:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01half_01_4_1a2febf53869fcfd71830c674652e7c7d4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kHALF</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4_1a996ff7940394a814ae164728312bf5ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4_1a10403a956a1d979e0cd10c8034e9f3e1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4_1a96b97cf0edb4ad984dd69278acfabd10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4_1af2839adf78453826b3a7f8422c9130fb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4_1a6d279398fbfedbbfffd0f7fbd1eb80b8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIbEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIbEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIbEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01bool_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIbEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:b:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01bool_01_4_1a590c40a56a5eabf6069340edc07d84e4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kBOOL</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4_1a60d418bc6cea9f845ee353d78f0c3d66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01void_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:voidP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01void_01_5_01_4_1a63d38011d7c02b6ce399d457537b1a93"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE" title="tensorrt_llm::runtime::BufferDataType::kTrtPointerType"><span class="n"><span class="pre">kTrtPointerType</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:TP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4_1ae513ee17ebefd7e2c538476736488aa1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::kUnderlyingType"><span class="n"><span class="pre">kUnderlyingType</span></span></a><span class="p"><span class="pre">.</span></span><span class="n"><span class="pre">getDataType</span></span><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::kUnderlyingType"><span class="n"><span class="pre">kUnderlyingType</span></span></a><span class="p"><span class="pre">.</span></span><span class="n"><span class="pre">isUnsigned</span></span><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE"></span><span id="tensorrt_llm::runtime::TRTDataType:TP:::kUnderlyingType__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4_1aafb13ea78bfe5c1ddf1a77093531b0ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnderlyingType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><span class="n"><span class="pre">TRTDataType</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferE"></span><span id="tensorrt_llm::runtime::IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="Link to this definition"></a><br /></dt>
<dd><p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref">tensorrt_llm::runtime::ITensor</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer9UniquePtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer9UniquePtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ab5684b52d1990c0a0f70c25d0b6c1465"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniquePtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer9SharedPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer9SharedPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2f0c3696d87a3887b38da126cbef1759"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a3892842dc7c9dcc1cad622470ac97999"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniqueConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a8eefcf151b57310d8ca78b8e25c46c39"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer8DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer8DataTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a4036935d2bbb442bf9d071df2eff24bd"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4dataEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="tensorrt_llm::runtime::IBuffer::data"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a1b29144573f04c11a8c59576a2ddca4f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="tensorrt_llm::runtime::IBuffer::dataC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a49f4130965beadb2b82a0b6bcd50cd11"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::data__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ada0b7259447dab20f65e144030f55cef"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to the underlying array at a given element index. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::data__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a797f41a093bb00596edaf60f329f66e1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to the underlying array at a given element index. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer7getSizeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getSizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ab280c88977e5cf7d3c3ab88e29dcbe62"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7getSizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the size (in number of elements) of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv"></span><span id="tensorrt_llm::runtime::IBuffer::getSizeInBytesC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a3121c0295ee572bb45067bba2d35defa"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizeInBytes</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the size (in bytes) of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer11getCapacityEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer11getCapacityEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer11getCapacityEv"></span><span id="tensorrt_llm::runtime::IBuffer::getCapacityC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a33c48c142ee4ee1c1537d1bef5a7ed16"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCapacity</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getCapacityEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the capacity of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getDataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a006dc512c6b37d582a2e825249c4a3a2"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the data type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer15getDataTypeNameEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer15getDataTypeNameEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer15getDataTypeNameEv"></span><span id="tensorrt_llm::runtime::IBuffer::getDataTypeNameC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a32794b7a1c08dfb83a19f78b8b388eaa"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getDataTypeName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer15getDataTypeNameEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getMemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ade5fc6e89a07ad03927a616fdcbb0463"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMemoryType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the memory type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer17getMemoryTypeNameEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer17getMemoryTypeNameEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer17getMemoryTypeNameEv"></span><span id="tensorrt_llm::runtime::IBuffer::getMemoryTypeNameC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1afd41586a8567409043799fdabca22dc4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getMemoryTypeName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer17getMemoryTypeNameEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::resize__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af1281049c93d0a1d61b8682170761ad4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">resize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">newSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7releaseEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7releaseEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7releaseEv"></span><span id="tensorrt_llm::runtime::IBuffer::release"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a51dc4a186d9b315dfe77aacd33677ff7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">release</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7releaseEv" title="Link to this definition"></a><br /></dt>
<dd><p>Releases the buffer. It will be reset to nullptr. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferD0Ev"></span><span id="tensorrt_llm::runtime::IBuffer::~IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1abeb460077884adc9aec0c351c9ef3637"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~IBuffer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer"></span><span id="tensorrt_llm::runtime::IBuffer::IBuffer__IBufferCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ac9fd5dfcd8486d8f8150fdb470c38055"></span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer" title="tensorrt_llm::runtime::IBuffer::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferaSERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferaSERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferaSERK7IBuffer"></span><span id="tensorrt_llm::runtime::IBuffer::assign-operator__IBufferCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a10744c55831bf12db698adbb511d660e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferaSERK7IBuffer" title="Link to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::slice__SharedPtr.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a6b9aed7e66f2bfc1f3f710d12dbceea6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Creates a sliced view on the underlying <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. The view will have the same data type as <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>buffer</strong> – The buffer to view. </p></li>
<li><p><strong>offset</strong> – The offset of the view. </p></li>
<li><p><strong>size</strong> – The size of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a40f5f57b98c9918b617c1de832390b9c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::slice__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a310d420f890b10b16951eae4b0227d21"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a9ffdc0f354482f983b6a16825ad25109"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr"></span><span id="tensorrt_llm::runtime::IBuffer::view__SharedPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a4fb6a8d4a92376cdea0957a26629f53b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> which can be independently resized. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>tensor</strong> – The tensor to view. </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::view__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a58bc0fcaa0356cf9e20ce1d06b16c70f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> with a different size. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>size</strong> – The size of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af28ec3097ffc66614052cef9392265bb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::wrap__voidP.DataType.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a6cf1157eb353c90dbe12711be9af63f2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Wraps the given <code class="docutils literal notranslate"><span class="pre">data</span></code> in an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code>. The <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> will not own the underlying <code class="docutils literal notranslate"><span class="pre">data</span></code> and cannot be resized beyond <code class="docutils literal notranslate"><span class="pre">capacity</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>data</strong> – The data to wrap. </p></li>
<li><p><strong>type</strong> – The data type of the <code class="docutils literal notranslate"><span class="pre">data</span></code>. </p></li>
<li><p><strong>size</strong> – The size of the buffer. </p></li>
<li><p><strong>capacity</strong> – The capacity of the buffer. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>An <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::wrap__voidP.DataType.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af1fb9e5497bfc31b149ff9477161068d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a1f5dd20e613af0bc2a05f0b09343535d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2e4240d0d7f78a278716c8faccf5e9f9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapERNSt6vectorI1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapERNSt6vectorI1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a84fdf2b484eee3440646edfece5b85b0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">v</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv"></span><span id="tensorrt_llm::runtime::IBuffer::memoryType__voidCP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2c714d66d0336cac0708008e59dd71cd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryType</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv" title="Link to this definition"></a><br /></dt>
<dd><p>Determine the memory type of a pointer. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7IBufferEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7IBufferEv"></span><span id="tensorrt_llm::runtime::IBuffer::IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ad6ad0acb873c47b64fc8fec029696ac2"></span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::toBytes__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1aade9d63190ac92fdb3561a61af5ab2ee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toBytes</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Returns an array index or size in bytes. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime11BufferRangeE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime11BufferRangeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferRange</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange10value_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange10value_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a577abddadeb9927120873f9d07325e37"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange9size_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange9size_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ac52032549a72c5078e9dea0d731626ab"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange9referenceE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange9referenceE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1acb95456868b359618379a83adc048eaf"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reference</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="tensorrt_llm::runtime::BufferRange::value_type"><span class="n"><span class="pre">value_type</span></span></a><span class="p"><span class="pre">&amp;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange15const_referenceE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange15const_referenceE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aed057be953bd7fb90a48f5a7c3ae1692"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_reference</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="tensorrt_llm::runtime::BufferRange::value_type"><span class="n"><span class="pre">value_type</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange7pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange7pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a286cbcbcb22a4f81104c9124130118c3"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange13const_pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange13const_pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aad04e26d81c56579e3d6a69ac5fa7412"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange8iteratorE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange8iteratorE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aeb69cf6a643d84e5c7d051bee4311869"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iterator</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE" title="tensorrt_llm::runtime::BufferRange::pointer"><span class="n"><span class="pre">pointer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange14const_iteratorE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange14const_iteratorE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a36e1f97c5d72f04e004f1624a5709986"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_iterator</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE" title="tensorrt_llm::runtime::BufferRange::const_pointer"><span class="n"><span class="pre">const_pointer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer"></span><span id="tensorrt_llm::runtime::BufferRange::BufferRange__IBufferR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a28e8d9ed5250f967aa8f06317074af7b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferRange</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5beginEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="tensorrt_llm::runtime::BufferRange::begin"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aadf490250d5ce134a93e32ee11a34980"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="tensorrt_llm::runtime::BufferRange::iterator"><span class="n"><span class="pre">iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">begin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5beginEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange3endEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="tensorrt_llm::runtime::BufferRange::end"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aefcb27b14d94348a6fad611d64fbac92"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="tensorrt_llm::runtime::BufferRange::iterator"><span class="n"><span class="pre">iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">end</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange3endEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange5beginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="tensorrt_llm::runtime::BufferRange::beginC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a7fd5ff6c02c395fa4237a0eb1d9330cf"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">begin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange5beginEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange3endEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="tensorrt_llm::runtime::BufferRange::endC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a758833712fb9f5aac0ec22591d767fee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">end</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange3endEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange6cbeginEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="tensorrt_llm::runtime::BufferRange::cbegin"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aad76274f379a648c172fa9f064f88d62"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cbegin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange6cbeginEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange4cendEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="tensorrt_llm::runtime::BufferRange::cend"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a28a237e06e8971046f0fd4d7d9011fff"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cend</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange4cendEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange6cbeginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="tensorrt_llm::runtime::BufferRange::cbeginC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a8e5dfa0b4e150b5fdc0ce78af3c5d243"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cbegin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange6cbeginEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange4cendEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="tensorrt_llm::runtime::BufferRange::cendC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a5df247c962064585ba1c47242a562dea"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cend</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4cendEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange4sizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange4sizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange4sizeEv"></span><span id="tensorrt_llm::runtime::BufferRange::sizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1add27991a49b2ff4b17b187711cd9eba6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4sizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRangeixE9size_type">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="tensorrt_llm::runtime::BufferRange::subscript-operator__size_type"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ab65060237344abbc8db90f2062abb3ab"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE" title="tensorrt_llm::runtime::BufferRange::reference"><span class="n"><span class="pre">reference</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">[]</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRangeixE9size_type" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRangeixE9size_type">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="tensorrt_llm::runtime::BufferRange::subscript-operator__size_typeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1acc8eff6f7541d117de3e09730425a430"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE" title="tensorrt_llm::runtime::BufferRange::const_reference"><span class="n"><span class="pre">const_reference</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">[]</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRangeixE9size_type" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5mDataE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5mDataE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5mDataE"></span><span id="tensorrt_llm::runtime::BufferRange::mData__TP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1afd541e9707bd79322c5290261787902e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">mData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mDataE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5mSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5mSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5mSizeE"></span><span id="tensorrt_llm::runtime::BufferRange::mSize__size_type"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ae18e3afe46fa34b6f3a8d5bfa7911bfc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="igptdecoderbatch-h">
<h2>iGptDecoderBatch.h<a class="headerlink" href="#igptdecoderbatch-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatchE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatchE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoderBatch</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="tensorrt_llm::runtime::IStatefulGptDecoder"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iGptDecoderBatch.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1GptDecoderBatch"><span class="std std-ref">tensorrt_llm::runtime::GptDecoderBatch</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a8968dd741ba1d28cddca68d850a62f59"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ac23bb332c54e6c2b665d5f35fc64baf8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1aefe802b00d9cc338c76f2bfc7bafad4c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::newRequest__SizeType.decoder_batch::RequestCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a795d229a59bc66688d9609ce62b40775"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="tensorrt_llm::runtime::decoder_batch::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Initialize the decoder at <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code> with a new <code class="docutils literal notranslate"><span class="pre">request</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forwardAsync__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a6c990b4aa463031ee4d4e2724997ae0e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TokenPtr"><span class="n"><span class="pre">TokenPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host process and return the token for synchronization. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forwardSync__decoder_batch::TokenCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a7171eb7513202c09360d4139117c424e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">token</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for the call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> associated with a token to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forward__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ab7e07276f33bfa6ef3c90baa49ee6eec"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests and wait for completion on the host. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a56bbbba5d310262f3614c9ca16d6e1d7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>batchIdx</strong> – index of the batch </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch8finalizeE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch8finalizeE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch8finalizeE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::finalize__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a37a0d386522f5b256addcfbe32005538"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch8finalizeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Gather final beam search results for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>. Result will only be available after event returned. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1aaa0b5e6c02b7043c3b29a1d8ca68df76"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize (actual)], marks finished requests (per batch) </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getCumLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1afa130a3df00b7d86e280f5caf017ab9d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getCumLogProbs__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a2176789e8a2e659f0b6fa4137dbffaaf"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[beamWidth], cumulative log probabilities (per beam) for request batchIdx, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a8b387a350a37f25a383bdcc18e66f8f8"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth, maxSeqLen], log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getLogProbs__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1af5635a23ba453db9f1e5edfa304e0958"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogProbs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getLogProbsE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[beamWidth, maxSeqLen], cumulative log probabilities (per beam) for request batchIdx, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getParentIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a1a37b811b0cc2a98507b57109844a06c"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParentIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getNbStepsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a935e778ac2c14276d940e09b9411e8ed"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbSteps</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::IGptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1abfc9e04a4f042f3c0220c84cff407ff3"></span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoderBatch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batchE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batchE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batchE"></span><span id="tensorrt_llm::runtime::decoder_batch"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1decoder__batch"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoder_batch</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch6OutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch6OutputE"></span><span class="target" id="iGptDecoderBatch_8h_1a575658a5bf5a41310da470e54bf965d9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7RequestE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7RequestE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a0f1d32403828dcdb0db0badc5d0ba8e6"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ConstTensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE" title="tensorrt_llm::runtime::ITensor::SharedConstPtr"><span class="n"><span class="pre">SharedConstPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a95d9dd610e7a6295d06f75a33cbb4c43"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9BufferPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request9BufferPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request9BufferPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a78becbf7fa5928420a994c0d9d9ddb65"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9BufferPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request7RequestE14ConstTensorPtr8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request7RequestE14ConstTensorPtr8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request7RequestE14ConstTensorPtr8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::Request__ConstTensorPtr.SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ac001937908b70ca7a843b86c06c1f7ee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::ConstTensorPtr"><span class="n"><span class="pre">ConstTensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">inputLen</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNewTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request7RequestE14ConstTensorPtr8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13decoder_batch7Request22generatedTokensPerStepEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13decoder_batch7Request22generatedTokensPerStepEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13decoder_batch7Request22generatedTokensPerStepEv"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::generatedTokensPerStepC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1afc321e118eab079b555525518a0d69d3"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generatedTokensPerStep</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13decoder_batch7Request22generatedTokensPerStepEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request3idsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::ids__ConstTensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ac8aaa11f8da8a1824edf1fb8d23ec40a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request14ConstTensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::ConstTensorPtr"><span class="n"><span class="pre">ConstTensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request3idsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request8inputLenE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request8inputLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request8inputLenE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::inputLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a88c0c0b993597c07f0dd9a941a04e741"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">inputLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request8inputLenE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::maxNewTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a5ab1bf54050a431e7c26df7451f2f0f1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request5endIdE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request5endIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request5endIdE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::endId__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ac1477f03a617c556bb264121200fcc0e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request5endIdE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request11draftTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request11draftTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request11draftTokensE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::draftTokens__BufferPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a1a524b5348e2d8ea442050f50ff3f1c7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9BufferPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::BufferPtr"><span class="n"><span class="pre">BufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">draftTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request11draftTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request11draftLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request11draftLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request11draftLogitsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::draftLogits__std::optional:TensorPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1afa9181bc52e25fe7dc0e925c69dc8d86"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">draftLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request11draftLogitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::embeddingBias__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1acbedf401f2f77534aa355f3ecbb0b755"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBias</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1af18cc1700b613308c14d492e69ab76e9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1addd0a66af57e7186287ac7e5853c0c9d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request18computeCumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request18computeCumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request18computeCumLogProbsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::computeCumLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ae608a0fadb52ad6ac8bd4ae96ab6f3bc"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeCumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request18computeCumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request15computeLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request15computeLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request15computeLogProbsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::computeLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ae35f498de3ce00f4ca94513b5c96ba8a"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request15computeLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1aa8152e055448762bd78ad70f53eda8ba"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE" title="tensorrt_llm::runtime::ITensor::SharedConstPtr"><span class="n"><span class="pre">SharedConstPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1ac182ee5aa51be63dfe20586ecaf40043"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEERKNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEERKNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEERKNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__std::vector:TensorConstPtr:CR.std::vector:b:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1aace73d69f6cbc9cb1342cd83ecf0c506"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorConstPtr"><span class="n"><span class="pre">TensorConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logits</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">active</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEERKNSt6vectorIbEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__std::vector:TensorConstPtr:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1aba21ef996e9e6fc1aca5bcc09fcd55ad"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorConstPtr"><span class="n"><span class="pre">TensorConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logits</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI14TensorConstPtrEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEERKNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEERKNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEERKNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__std::vector:TensorPtr:CR.std::vector:b:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1ad2b3fc875250c83be03032f984154830"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logits</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">active</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEERKNSt6vectorIbEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__std::vector:TensorPtr:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1ad5c9b6c5a35df62813d6bedaae41660f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logits</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputERKNSt6vectorI9TensorPtrEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6logitsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input6logitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input6logitsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::logits__std::vector:TensorConstPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a9a9e759d4b5fa7e52ebded01ca1479e2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorConstPtr"><span class="n"><span class="pre">TensorConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6logitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6activeE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input6activeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input6activeE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::active__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a85bb8bda589fcebf0c19994d76a26968"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">active</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6activeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::cacheIndirection__TensorConstPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a29217fe46691aef9e896775d4aa30c83"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input14TensorConstPtrE" title="tensorrt_llm::runtime::decoder_batch::Input::TensorConstPtr"><span class="n"><span class="pre">TensorConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input16cacheIndirectionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Token</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::Token__CudaEventRR.std::vector:b:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1ab35d80ddc444d443a35ae642d3e9f862"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Token</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">active</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5eventE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token5eventE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token5eventE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::event__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1a8a4fa5d069ee58377e79ddde0955dbde"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">event</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5eventE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token6activeE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token6activeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token6activeE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::active__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1a92563b2a16378f689f9d600d8712f8fd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">active</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token6activeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="istatefulgptdecoder-h">
<h2>iStatefulGptDecoder.h<a class="headerlink" href="#istatefulgptdecoder-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoderE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iStatefulGptDecoder.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch"><span class="std std-ref">tensorrt_llm::runtime::IGptDecoderBatch</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a310eb1dafd035c945176ea533ec74588"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a3987d8ce3a11babf865981af9d4ea18e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::setup__SizeType.SizeType.SizeType.SizeType.SizeType.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a55b54fee12d6a2b3c7951b3d57d06aca"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxKvCacheLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokensPerStep</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>Setup the decoder before calling <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a152618c3967cc29d314bd77d17c51e27"><span class="std std-ref"><span class="pre">forward()</span></span></a></code>, also calls reshapeBuffers. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::newBatch__GenerationInputCR.GenerationOutputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a2b8ead8e3fca4dcb2c6e95d8e063b3cf"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK16GenerationOutputRK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Initialize the decoder with new batch of inputs. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::forwardAsync__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a7bb51780e6e9a6c41de214ffac8b07be"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host thread. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder11forwardSyncEv">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder11forwardSyncEv"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder11forwardSyncEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::forwardSync"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a2a2a7296d9137e434da174a04626722a"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder11forwardSyncEv" title="Link to this definition"></a><br /></dt>
<dd><p>Wait for the last call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::forward__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a152618c3967cc29d314bd77d17c51e27"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE" title="Link to this definition"></a><br /></dt>
<dd><p>Run one step for all requests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder8finalizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder8finalizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder8finalizeEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::finalizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1ae29a2934bf1c89392862b8d83f199fa0"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder8finalizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Gather final beam search results for all requests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a3f98067a78b64a319acb4a8b667e6249"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth, maxSequenceLength], all token ids, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder14getCumLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder14getCumLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder14getCumLogProbsEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getCumLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1ad0a22a269ec223d875f0d68bb5efc744"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder14getCumLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder11getLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder11getLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder11getLogProbsEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1ad1d6dd18c64d32cf98b31792c9fc499b"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder11getLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensE8SizeType"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getNewTokens__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1aff1095cf556ce2d4893a43fbc0c5527b"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNewTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iter</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Get tokens generated in one step of last forward pass. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>iter</strong> – The iteration within [0; maxTokensPerStep) for which to get the tokens </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>[batchSize, beamWidth], tokens generated in <code class="docutils literal notranslate"><span class="pre">iter</span></code> (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder15getAllNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder15getAllNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder15getAllNewTokensEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getAllNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a9ebd85b4c95c5bb2623565e4ab3983b6"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getAllNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder15getAllNewTokensEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get maxTokensPerStep tokens generated in the last forward pass. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxTokensPerStep, batchSize, maxBeamWidth], tokens generated in last forward pass, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getNbFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a551e560fe7c9120c500f47fa84514d99"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[1], number of finished sequences, in pinned host memory </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoderD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoderD0Ev"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::~IStatefulGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a7dd550e0f0d980fd3f3f655f358e8ff0"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~IStatefulGptDecoder</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::IStatefulGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a898fa69e15fa0d26fde3c7f50fdb8881"></span><span class="sig-name descname"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoderE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoderE"></span><span id="tensorrt_llm::runtime::decoder"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1decoder"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5InputE"></span><span id="tensorrt_llm::runtime::decoder::Input"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1a00dc3503cac2813698c97486fba9ca51"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr"></span><span id="tensorrt_llm::runtime::decoder::Input::Input__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1adbf9a04fb7b5c83766b236df5060d24a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input6logitsE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input6logitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input6logitsE"></span><span id="tensorrt_llm::runtime::decoder::Input::logits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1a8b92e3c3e42f408503691dd5fb51e79e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input6logitsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::decoder::Input::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1ae425ecaffbd89893ed70d3ad32c660b1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6OutputE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6OutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6OutputE"></span><span id="tensorrt_llm::runtime::decoder::Output"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1a2c4dff939d33fec2fd199699c216ff28"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output6OutputEv">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output6OutputEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output6OutputEv"></span><span id="tensorrt_llm::runtime::decoder::Output::Output"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1af52215c47e70f6af33b6353883c24948"></span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output6OutputEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::decoder::Output::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1ae9e66b08bf05167a9d16130d0518fc0e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="tensorrt_llm::runtime::decoder::Output::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE"></span><span id="tensorrt_llm::runtime::decoder::Output::sequenceLengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1a25913f9ceca02070a650d1135ba87a72"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="tensorrt_llm::runtime::decoder::Output::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="itensor-h">
<h2>iTensor.h<a class="headerlink" href="#itensor-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv48nvinfer1">
<span id="_CPPv38nvinfer1"></span><span id="_CPPv28nvinfer1"></span><span id="nvinfer1"></span><span class="target" id="namespacenvinfer1"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">nvinfer1</span></span></span><a class="headerlink" href="#_CPPv48nvinfer1" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.ITensor::ShapeCR"></span><span class="target" id="iTensor_8h_1ab843bef84b07b782c05c682399ee2b1a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to print a shape. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.ITensorCR"></span><span class="target" id="iTensor_8h_1a2a38821feb0276616dde2d32020ef451"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to print a tensor with its shape. </p>
</dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensorE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensorE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensorE"></span><span id="tensorrt_llm::runtime::ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9UniquePtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9UniquePtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a0ebcd2ee37abad2960daf1346816f348"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniquePtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9SharedPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9SharedPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a31e67140fadaeb6eaaf096ff37d640ad"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ad199f95dd25dfbef298cc7d1f3655d66"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniqueConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor14SharedConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor14SharedConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a591e6b0aa01a0043967262ffdcc3974d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5ShapeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab2e78d36e5cc72b4443e832a46147fcc"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7DimTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7DimTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7DimTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a08933f77dd49cd3f0ff4bf832411d6f8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DimType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_reference_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">decltype</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">d</span></span><span class="p"><span class="pre">[</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7DimTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensorD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensorD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensorD0Ev"></span><span id="tensorrt_llm::runtime::ITensor::~ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a857da95b81330009b368308ed0c2d3d9"></span><span class="sig-name descname"><span class="n"><span class="pre">~ITensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7ITensor8getShapeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7ITensor8getShapeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7ITensor8getShapeEv"></span><span id="tensorrt_llm::runtime::ITensor::getShapeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a9f2d4a0903e581784f5ed94d8ed1d58d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getShape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor8getShapeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the tensor dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::reshape__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a772377264b089406fb1cf743ac3163f3"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reshape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Sets the tensor dimensions. The new size of the tensor will be <code class="docutils literal notranslate"><span class="pre">volume(dims)</span></code></p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor6resizeENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor6resizeENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor6resizeENSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::resize__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a8ed7ae223a11d8d516216a4d28286f62"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">resize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">newSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor6resizeENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor"></span><span id="tensorrt_llm::runtime::ITensor::ITensor__ITensorCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ade4b1d2a214d378f73255357e32bee1f"></span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor" title="tensorrt_llm::runtime::ITensor::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor" title="Link to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensoraSERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensoraSERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensoraSERK7ITensor"></span><span id="tensorrt_llm::runtime::ITensor::assign-operator__ITensorCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a943f0acaa8c746fa85980f4e13512a2c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensoraSERK7ITensor" title="Link to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::squeeze__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a5ca07f1cc675136634c6a435b5b8f8f3"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">squeeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Removes the given <em>unit</em> dimensions from this tensor. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9unsqueezeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9unsqueezeE8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::unsqueeze__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a9725635429cc853be302ecee91bd4662"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">unsqueeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeE8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Adds a <em>unit</em> dimension at the specified position. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERK5Shape">
<span id="_CPPv3NK12tensorrt_llm7runtime7ITensor11shapeEqualsERK5Shape"></span><span id="_CPPv2NK12tensorrt_llm7runtime7ITensor11shapeEqualsERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::shapeEquals__ShapeCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab84847a01479731500363a637003e03a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shapeEquals</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERK5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERKNSt16initializer_listI8SizeTypeEE">
<span id="_CPPv3NK12tensorrt_llm7runtime7ITensor11shapeEqualsERKNSt16initializer_listI8SizeTypeEE"></span><span id="_CPPv2NK12tensorrt_llm7runtime7ITensor11shapeEqualsERKNSt16initializer_listI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::ITensor::shapeEquals__std::initializer_list:SizeType:CRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a0589c32d94c01c24f0f08a313aeab0ba"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shapeEquals</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">initializer_list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor11shapeEqualsERKNSt16initializer_listI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEbPK1T8SizeType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEPK1T8SizeType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEPK1T8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab3064d90488eb76a945df538a48ba683"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shapeEquals</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEbPK1T8SizeType" title="tensorrt_llm::runtime::ITensor::shapeEquals::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">count</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime7ITensor11shapeEqualsEbPK1T8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor6volumeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor6volumeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor6volumeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::volume__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a2afffa26551b43982bf51fa4920d4ade"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">volume</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor6volumeERK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the volume of the dimensions. Returns -1 if <code class="docutils literal notranslate"><span class="pre">d.nbDims</span> <span class="pre">&lt;</span> <span class="pre">0</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::volumeNonNegative__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aac34c9309ce068d8216a54343348e21d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">volumeNonNegative</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the volume of the dimensions. Throws if <code class="docutils literal notranslate"><span class="pre">d.nbDims</span> <span class="pre">&lt;</span> <span class="pre">0</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::squeeze__ShapeCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a7c9ddd4f8b1eaa7b1a6276c49b2c8672"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">squeeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Removes the given <em>unit</em> dimension from <code class="docutils literal notranslate"><span class="pre">shape</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape to squeeze. </p></li>
<li><p><strong>dim</strong> – The dimension that should be removed (“squeezed”). </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A new shape without the unit dimension. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeERK5Shape8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9unsqueezeERK5Shape8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9unsqueezeERK5Shape8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::unsqueeze__ShapeCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a637c76757c7e71c448ec24217de99e70"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">unsqueeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9unsqueezeERK5Shape8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>Add a <em>unit</em> dimension to <code class="docutils literal notranslate"><span class="pre">shape</span></code> at the specified position. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape to unsqueeze. </p></li>
<li><p><strong>dim</strong> – The dimension where unit dimension should be added. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A new shape with the added unit dimension. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::slice__SharedPtr.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aa22bf50eff27dbfc64eed003851f4778"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Creates a sliced view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. The view will have the same data type as <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>offset</strong> – The offset of the view w.r.t. dimension 0 of the tensor. </p></li>
<li><p><strong>size</strong> – The size of the view w.r.t. dimension 0 of the tensor. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a201a60f7da459ec35f5d750e8d7053c4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::slice__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a3bc0148d89f892df6beff0f63ea33e6a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a01d828a32b3f280c1ee64ce5211da20b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::view__IBuffer::SharedPtr.ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aeb7a31c92f9ebc17e4648ba4fff7e16e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">buffer</span></code> (or tensor) with the given shape. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>shape</strong> – The shape of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewERR9TConstPtrRK5Shape"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewERR9TConstPtrRK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="tensorrt_llm::runtime::ITensor::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a481831d7c0d45fe462e91b07beef2617"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="tensorrt_llm::runtime::ITensor::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr"></span><span id="tensorrt_llm::runtime::ITensor::view__SharedPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a0d748a6fdb8219af29ba918878bf44b2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> which can be independently reshaped. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>tensor</strong> – The tensor to view. </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::wrap__voidP.nvinfer1::DataType.ShapeCR.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1abb208b99fc6a404a13215fe2d43637a4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd><p>Wraps the given <code class="docutils literal notranslate"><span class="pre">data</span></code> in an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code>. The <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> will not own the underlying <code class="docutils literal notranslate"><span class="pre">data</span></code> and cannot be reshaped beyond <code class="docutils literal notranslate"><span class="pre">capacity</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>data</strong> – The data to wrap. </p></li>
<li><p><strong>type</strong> – The data type of the <code class="docutils literal notranslate"><span class="pre">data</span></code>. </p></li>
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>capacity</strong> – The capacity of the buffer. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>An <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::wrap__voidP.nvinfer1::DataType.ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a5fdde0b2cc5bb0ee07c042964bb48c85"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5ShapeNSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5ShapeNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a6cb1dc1fe13779d7d0bed2062df2c0e0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a496e23f8dcc29f792a7e0f70fca11973"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapERNSt6vectorI1TEERK5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapERNSt6vectorI1TEERK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab27a7a0ae81205699d2b2bc618367857"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">v</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::ITensor::makeShape__std::initializer_list:SizeType:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a7ba94f17049ee1d6048ae69be48ecb67"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">makeShape</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">initializer_list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd><p>A convenience function to create a tensor shape with the given dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor8toStringERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor8toStringERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor8toStringERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::toString__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1af9ba06b8cbf144ea573e6767af85bd43"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor8toStringERK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>A convenience function for converting a tensor shape to a <code class="docutils literal notranslate"><span class="pre">string</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapeRK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapeRK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapeRK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::shapeEquals__ShapeCR.ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ae39159cc236f70454983d0193570e1c4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shapeEquals</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lhs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">rhs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapeRK5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>A convenience function to compare shapes. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsEbRK5ShapePK1T8SizeType">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapePK1T8SizeType"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsERK5ShapePK1T8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aac424f6ed20794f38cbd59006f96c57a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shapeEquals</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lhs</span></span>, <a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsEbRK5ShapePK1T8SizeType" title="tensorrt_llm::runtime::ITensor::shapeEquals::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">count</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor11shapeEqualsEbRK5ShapePK1T8SizeType" title="Link to this definition"></a><br /></dt>
<dd><p>A convenience function to compare shapes. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorEv">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7ITensorEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7ITensorEv"></span><span id="tensorrt_llm::runtime::ITensor::ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a60742a4752e9d9d5a82fff9e24518953"></span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-static-functions">Protected Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor8castSizeE6size_t">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor8castSizeE6size_t"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor8castSizeE6size_t"></span><span id="tensorrt_llm::runtime::ITensor::castSize__s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a8966794543216594dab695f03d1e6c9e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7DimTypeE" title="tensorrt_llm::runtime::ITensor::DimType"><span class="n"><span class="pre">DimType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">castSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">newSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor8castSizeE6size_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="ipcutils-h">
<h2>ipcUtils.h<a class="headerlink" href="#ipcutils-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb">
<span id="_CPPv3N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb"></span><span id="_CPPv2N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb"></span><span id="tensorrt_llm::runtime::setPeerAccess__WorldConfig.b"></span><span class="target" id="ipcUtils_8h_1a16050a9ed921a12d13a273478086f95c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPeerAccess</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enable</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemoryE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemoryE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemoryE"></span><span id="tensorrt_llm::runtime::IpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IpcMemory</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a1c17fa36835ec34cb3cef927e85967eb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE"></span><span id="tensorrt_llm::runtime::IpcMemory::IpcMemory__WorldConfig.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1ab60128a164106fe08d754357214b5b82"></span><span class="sig-name descname"><span class="n"><span class="pre">IpcMemory</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">bufferSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemoryD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemoryD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemoryD0Ev"></span><span id="tensorrt_llm::runtime::IpcMemory::~IpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a0f068272bfaeadb2e976b44adf47c484"></span><span class="sig-name descname"><span class="n"><span class="pre">~IpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv"></span><span id="tensorrt_llm::runtime::IpcMemory::getCommPtrsTensorC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a8bcbdce69d5e3de95a0f742e430f822c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getCommPtrsTensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE"></span><span id="tensorrt_llm::runtime::IpcMemory::FLAGS_SIZE__s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a4146075f3f6beba4eab4a765506be9bd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FLAGS_SIZE</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">kernels</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MAX_ALL_REDUCE_BLOCKS</span></span><span class="w"> </span><span class="o"><span class="pre">*</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><span class="n"><span class="pre">uint32_t</span></span><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv"></span><span id="tensorrt_llm::runtime::IpcMemory::allocateIpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a68e4ab83a9194bdd5cf59a1d4ee68a80"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocateIpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv"></span><span id="tensorrt_llm::runtime::IpcMemory::destroyIpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a7b7c035819f96cb37702472ea179c33b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">destroyIpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE"></span><span id="tensorrt_llm::runtime::IpcMemory::mWorldConfig__WorldConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1aab66f175e30581b311c654816aa85ea4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE"></span><span id="tensorrt_llm::runtime::IpcMemory::mCommPtrs__std::vector:voidP:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1aa15ac983a61b79976435b4d067daccc2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommPtrs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE"></span><span id="tensorrt_llm::runtime::IpcMemory::mBufferSize__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a95b1167806fe77f1f22810ac7477d27d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBufferSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE"></span><span id="tensorrt_llm::runtime::IpcMemory::mBufferPtr__voidP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1adf62ec2d27e1382c5414188fd3982f8b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">mBufferPtr</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="memorycounters-h">
<h2>memoryCounters.h<a class="headerlink" href="#memorycounters-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCountersE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCountersE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCountersE"></span><span id="tensorrt_llm::runtime::MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryCounters</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a5aa02d808601c67ba934f8a0a9534d6e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ab5afaef89516ec27d7934229de37e387"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DiffType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ptrdiff_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a97911bd1f11691ed4a652bf78e049e08"></span><span class="sig-name descname"><span class="n"><span class="pre">MemoryCounters</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getGpuC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a09119207f2dfca61e4e7bd5a66c36654"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpu</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getCpuC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1af643b8967c9f99d8efd647d7b3efe281"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCpu</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getPinnedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ae37c0265530ae4fac20d31da5712d3c0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPinned</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getGpuDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a399a7926f7819096b4f29fa04b3290de"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpuDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getCpuDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a62f4e6d3f0dd20c4ec6b35b4b6df0d47"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCpuDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getPinnedDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a4571ac85d01e0844b946d57615523ad0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPinnedDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateEv8SizeType">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateE8SizeType"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateE8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a891aab52104e74b95138f4bf20c111c2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateEv8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType"></span><span id="tensorrt_llm::runtime::MemoryCounters::allocate__MemoryType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a0b437c35c4af66d2f2b2379a40c136d6"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateEv8SizeType">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateE8SizeType"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateE8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a6301480be8d33d2a3e7dbd54119467c9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deallocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateEv8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType"></span><span id="tensorrt_llm::runtime::MemoryCounters::deallocate__MemoryType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a4cbe26fb0d89352c1de2cb6f63220aa2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deallocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters8toStringEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters8toStringEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters8toStringEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::toStringC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ad6981e0fc09bbeefdf4b2854fd03d77a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toString</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters8toStringEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getInstance"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a5fd849b9f6dd583910a3aa4acb8ea391"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="tensorrt_llm::runtime::MemoryCounters"><span class="n"><span class="pre">MemoryCounters</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getInstance</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei"></span><span id="tensorrt_llm::runtime::MemoryCounters::bytesToString__SizeType.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ad3e1b845f41a4245556557b1393881f9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">bytesToString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bytes</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei"></span><span id="tensorrt_llm::runtime::MemoryCounters::bytesToString__DiffType.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ae644945f3b01297e95a0f1724b5b4d8a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">bytesToString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bytes</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mGpuE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters4mGpuE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters4mGpuE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mGpu__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a641ecc1e58aa222f4a7e269aa589e70f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpu</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mGpuE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mCpuE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters4mCpuE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters4mCpuE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mCpu__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a27771c9783292c51de1b8a59aca33d71"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCpu</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mCpuE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters7mPinnedE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters7mPinnedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters7mPinnedE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mPinned__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a7eaaa821b0f7cbf8f1a2264e5f1b1c25"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPinned</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters7mPinnedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mGpuDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a091075678e2a77da992d619ccc27ee69"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpuDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mCpuDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a55b97586df47d83ece390da5b9da7ace"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCpuDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mPinnedDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a3ac02c1fc74dd309e7680be2c27ecbf0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPinnedDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters9mInstanceE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters9mInstanceE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters9mInstanceE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mInstance__MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a6de4abe7564f08c88fedabcff29ffeca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">thread_local</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="tensorrt_llm::runtime::MemoryCounters"><span class="n"><span class="pre">MemoryCounters</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInstance</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters9mInstanceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="samplingconfig-h">
<h2>samplingConfig.h<a class="headerlink" href="#samplingconfig-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfigE"></span><span id="tensorrt_llm::runtime::SamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType"></span><span id="tensorrt_llm::runtime::SamplingConfig::SamplingConfig__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a1b84a281c4413d5a4df922e9074d85ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9beamWidthE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9beamWidthE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9beamWidthE"></span><span id="tensorrt_llm::runtime::SamplingConfig::beamWidth__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a871887c0857fa88f45fadcd659c20824"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9beamWidthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig11temperatureE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig11temperatureE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig11temperatureE"></span><span id="tensorrt_llm::runtime::SamplingConfig::temperature__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1aa0234e25caafb8f7e2540e635354f1a7"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">temperature</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig11temperatureE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9minLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9minLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9minLengthE"></span><span id="tensorrt_llm::runtime::SamplingConfig::minLength__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1aacc380bc9c6ff46fd2edd112b1eac9c0"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">minLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9minLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::repetitionPenalty__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ab267d53af1ff989a2cda73069dfd42fa"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">repetitionPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::presencePenalty__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1af40f62527a6a93da70def3daafdc8001"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">presencePenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topKE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig4topKE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig4topKE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topK__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ab1588865e1c6e97bfebfe12b7ee34e37"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topK</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topKE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topPE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig4topPE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig4topPE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topP__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a6d960ae9fb60fa44c616cf4b16a6977d"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topP</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topPE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig10randomSeedE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig10randomSeedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig10randomSeedE"></span><span id="tensorrt_llm::runtime::SamplingConfig::randomSeed__OptVec:unsigned-l-l:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a6468682988b1992a5be330aaadc9641e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">long</span></span><span class="w"> </span><span class="kt"><span class="pre">long</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">randomSeed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig10randomSeedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9topPDecayE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9topPDecayE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9topPDecayE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPDecay__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ad25e0aec9100bf05d100885677dfaa03"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPDecay</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9topPDecayE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig7topPMinE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig7topPMinE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig7topPMinE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPMin__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a2f772d5c0be4d9fe80fdf0e142a731d5"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPMin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig7topPMinE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPResetIds__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ae2a3d686956933a6938925309496a33e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPResetIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE"></span><span id="tensorrt_llm::runtime::SamplingConfig::beamSearchDiversityRate__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a58a44a42e3086649b6b510222b007ac6"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamSearchDiversityRate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::lengthPenalty__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a393378cbfb39e5a147b88a8601050947"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengthPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig24draftAcceptanceThresholdE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig24draftAcceptanceThresholdE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig24draftAcceptanceThresholdE"></span><span id="tensorrt_llm::runtime::SamplingConfig::draftAcceptanceThreshold__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a914d673110a9a51924ec03567f2b6fb5"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">draftAcceptanceThreshold</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig24draftAcceptanceThresholdE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a78311bb676349c17d54fee63f3d54396"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FloatType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a82ed7012baf2949351e80937329b530a"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OptVec</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="tllmlogger-h">
<h2>tllmLogger.h<a class="headerlink" href="#tllmlogger-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLoggerE">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLoggerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLoggerE"></span><span id="tensorrt_llm::runtime::TllmLogger"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TllmLogger</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLoggerE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE"></span><span id="tensorrt_llm::runtime::TllmLogger::log__Severity.nvinfer1::AsciiCharCP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1a052c775ee14bc0d741d26d28c5b3f311"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">log</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">severity</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">AsciiChar</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">msg</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger8getLevelEv">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger8getLevelEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger8getLevelEv"></span><span id="tensorrt_llm::runtime::TllmLogger::getLevel"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1ae6b565ac5ee12cfcd305c0f2c0bd4b1e"></span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLevel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8getLevelEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity"></span><span id="tensorrt_llm::runtime::TllmLogger::setLevel__Severity"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1a02ca7ebe0eec266f8b6ab4b66e9f0275"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLevel</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">level</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="worldconfig-h">
<h2>worldConfig.h<a class="headerlink" href="#worldconfig-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfigE"></span><span id="tensorrt_llm::runtime::WorldConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">WorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::WorldConfig::WorldConfig__SizeType.SizeType.SizeType.SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a24619a7cd8eab9da4347680e248b922a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">WorldConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">rank</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig7getSizeEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab015227c42d7122c0e33f22e03ac8228"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getTensorParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a389ba0bc53cd7fe0259bdb2124ee156f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isTensorParallelCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1af3627cbcb19d1519338fea2e43335633"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTensorParallel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab90953f5e50d6fdef2d3c76ad8295f98"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isPipelineParallelCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a37ec8744e7dc7ff6211c319a33ecc628"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isPipelineParallel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig7getRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig7getRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab99195015c992ba7a735c7880cc62bb6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getRankEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getGpusPerNodeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a32f320679d97d1a278e84bb21af89494"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpusPerNode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getDeviceCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a35f8f78f02108160d180c0b70f78b414"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a619d3cbd60a9cad3d0232844e8693739"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getTensorParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab00da06440a9b4f3305d9811dcd7b61b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isFirstPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a6a5a2512cce31560ae828d7fd1968c22"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFirstPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isLastPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a7e9cc05c32f9a1edf82d4017e34a3bef"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLastPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv" title="Link to this definition"></a><br /></dt>
<dd><p>Is my rank the last rank in its pipeline? </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig11getLastRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig11getLastRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig11getLastRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getLastRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a64bb69b0f709d7081da9201c9990457c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLastRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig11getLastRankEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelGroupC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a79368880ccca35154c16cbf409c97430"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelGroup</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::WorldConfig::validConfig__nvinfer1::ILoggerR.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab34b3cafb821617e539a8d736c09e3f5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">validConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logger</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::WorldConfig::mpi__nvinfer1::ILoggerR.SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ad8ec600840b45582dccbc01eb7926ff0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logger</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::WorldConfig::mpi__SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab0651f2698fe41426584b3ab69529460"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE"></span><span id="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a2d68f68654533c7b0b9be92dae183082"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE"></span><span id="tensorrt_llm::runtime::WorldConfig::mTensorParallelism__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a60d3f95841a56216582ff4d1749841db"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensorParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE"></span><span id="tensorrt_llm::runtime::WorldConfig::mPipelineParallelism__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a455c07380f4ab9f5d299de40011d03be"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig5mRankE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig5mRankE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig5mRankE"></span><span id="tensorrt_llm::runtime::WorldConfig::mRank__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a834b8b05fadab2839ad94c0389385330"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRank</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig5mRankE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE"></span><span id="tensorrt_llm::runtime::WorldConfig::mGpusPerNode__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a3c6d093540443a20a3204d0129d8be79"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpusPerNode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="prompttuningparams-h">
<h2>promptTuningParams.h<a class="headerlink" href="#prompttuningparams-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TTensor</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericPromptTuningParams</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1a7f0226e537f0c4a164c298adf9731e67"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TTensor"><span class="n"><span class="pre">TTensor</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1a1dc6277cb0a7c8768f9b3e7609189c8c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams25GenericPromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams25GenericPromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams25GenericPromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::GenericPromptTuningParams::GenericPromptTuningParams__TensorPtr.TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1a08b54a1a315a75cada66405e31709668"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenericPromptTuningParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingTable</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tasks</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams25GenericPromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams14embeddingTableE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams14embeddingTableE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams14embeddingTableE"></span><span id="tensorrt_llm::runtime::GenericPromptTuningParams::embeddingTable__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1a718fcc98779017166cbe0a748c081414"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingTable</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams14embeddingTableE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams5tasksE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams5tasksE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams5tasksE"></span><span id="tensorrt_llm::runtime::GenericPromptTuningParams::tasks__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1aea7aafec3a272782a1a467eb159af6e1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tasks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams5tasksE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9vocabSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams9vocabSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams9vocabSizeE"></span><span id="tensorrt_llm::runtime::GenericPromptTuningParams::vocabSize__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1acb732cd787508ea7013cf1e9002590ea"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::GenericPromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">vocabSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams9vocabSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams19promptTuningEnabledE">
<span id="_CPPv3N12tensorrt_llm7runtime25GenericPromptTuningParams19promptTuningEnabledE"></span><span id="_CPPv2N12tensorrt_llm7runtime25GenericPromptTuningParams19promptTuningEnabledE"></span><span id="tensorrt_llm::runtime::GenericPromptTuningParams::promptTuningEnabled__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenericPromptTuningParams_1a4f994aa4122125396dee70846aec1318"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">promptTuningEnabled</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams19promptTuningEnabledE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime18PromptTuningParamsE">
<span id="_CPPv3N12tensorrt_llm7runtime18PromptTuningParamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime18PromptTuningParamsE"></span><span id="tensorrt_llm::runtime::PromptTuningParams"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1PromptTuningParams"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningParams</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE" title="tensorrt_llm::runtime::GenericPromptTuningParams"><span class="n"><span class="pre">GenericPromptTuningParams</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParamsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1PromptTuningParams_1acf57960de488b9fcc3021a34690f003a"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1PromptTuningParams_1a1d6aab181a91c9a97b57429f08b53b55"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime25GenericPromptTuningParamsE" title="tensorrt_llm::runtime::GenericPromptTuningParams"><span class="n"><span class="pre">GenericPromptTuningParams</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime25GenericPromptTuningParams8SizeTypeE" title="tensorrt_llm::runtime::GenericPromptTuningParams::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime18PromptTuningParams18PromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime18PromptTuningParams18PromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime18PromptTuningParams18PromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::PromptTuningParams::PromptTuningParams__TensorPtr.TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1PromptTuningParams_1a6439bfa322d7dc1f589cd3877640a952"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::PromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingTable</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::PromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tasks</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::PromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams18PromptTuningParamsE9TensorPtr9TensorPtr9TensorPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime18PromptTuningParams15fillTasksTensorE9TensorPtrK8SizeTypeK8SizeTypeRKNSt6vectorI8SizeTypeEERKNSt6vectorI8SizeTypeEERK13BufferManagerb">
<span id="_CPPv3N12tensorrt_llm7runtime18PromptTuningParams15fillTasksTensorE9TensorPtrK8SizeTypeK8SizeTypeRKNSt6vectorI8SizeTypeEERKNSt6vectorI8SizeTypeEERK13BufferManagerb"></span><span id="_CPPv2N12tensorrt_llm7runtime18PromptTuningParams15fillTasksTensorE9TensorPtrK8SizeTypeK8SizeTypeRKNSt6vectorI8SizeTypeEERKNSt6vectorI8SizeTypeEERK13BufferManagerb"></span><span id="tensorrt_llm::runtime::PromptTuningParams::fillTasksTensor__TensorPtr.SizeTypeC.SizeTypeC.std::vector:SizeType:CR.std::vector:SizeType:CR.BufferManagerCR.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1PromptTuningParams_1a95c1342b784bdf139c4d0f0502851735"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">fillTasksTensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams9TensorPtrE" title="tensorrt_llm::runtime::PromptTuningParams::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tasksHost</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE" title="tensorrt_llm::runtime::PromptTuningParams::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE" title="tensorrt_llm::runtime::PromptTuningParams::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numContextRequests</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE" title="tensorrt_llm::runtime::PromptTuningParams::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">reqBeamWidths</span></span>, <span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams8SizeTypeE" title="tensorrt_llm::runtime::PromptTuningParams::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">reqPromptLengths</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">packedInput</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime18PromptTuningParams15fillTasksTensorE9TensorPtrK8SizeTypeK8SizeTypeRKNSt6vectorI8SizeTypeEERKNSt6vectorI8SizeTypeEERK13BufferManagerb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="../python-api/tensorrt_llm.runtime.html" class="btn btn-neutral float-left" title="Runtime" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="../blogs/H100vsA100.html" class="btn btn-neutral float-right" title="H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token." accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2023, NVidia.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink