kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
cb04a921a1
TensorRT-LLMs/_cpp_gen/runtime.html
石晓伟 cb04a921a1
correction of typo (#115)
2023-10-25 19:55:42 +08:00

4625 lines
953 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Runtime &mdash; tensorrt_llm documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js?v=b3ba4146"></script>
<script src="../_static/doctools.js?v=888ff710"></script>
<script src="../_static/sphinx_highlight.js?v=4825356b"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="prev" title="Runtime" href="../python-api/tensorrt_llm.runtime.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
tensorrt_llm
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">TensorRT-LLM Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../gpt_runtime.html">C++ GPT Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../batch_manager.html">The Batch Manager in TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../gpt_attention.html">Multi-head, Multi-query and Group-query Attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="../precision.html">Numerical Precision</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance.html">Performance of TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation.html">Build From Sources</a></li>
<li class="toctree-l1"><a class="reference internal" href="../2023-05-19-how-to-debug.html">How to debug</a></li>
<li class="toctree-l1"><a class="reference internal" href="../2023-05-17-how-to-add-a-new-model.html">How to add a new model</a></li>
<li class="toctree-l1"><a class="reference internal" href="../graph-rewriting.html">Graph Rewriting Module</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Python API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.layers.html">Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.functional.html">Functionals</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.models.html">Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.plugin.html">Plugin</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.quantization.html">Quantization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.runtime.html">Runtime</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">C++ API</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Runtime</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#buffermanager-h">bufferManager.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#common-h">common.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#cudaevent-h">cudaEvent.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#cudastream-h">cudaStream.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#decodinginput-h">decodingInput.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#decodingoutput-h">decodingOutput.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#generationinput-h">generationInput.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#generationoutput-h">generationOutput.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#gptdecoder-h">gptDecoder.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#gptdecoderbatch-h">gptDecoderBatch.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#gptjsonconfig-h">gptJsonConfig.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#gptmodelconfig-h">gptModelConfig.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#gptsession-h">gptSession.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#ibuffer-h">iBuffer.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#igptdecoderbatch-h">iGptDecoderBatch.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#istatefulgptdecoder-h">iStatefulGptDecoder.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#itensor-h">iTensor.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#ipcutils-h">ipcUtils.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#memorycounters-h">memoryCounters.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#samplingconfig-h">samplingConfig.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#tllmlogger-h">tllmLogger.h</a></li>
<li class="toctree-l2"><a class="reference internal" href="#worldconfig-h">worldConfig.h</a></li>
</ul>
</li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">tensorrt_llm</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Runtime</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/_cpp_gen/runtime.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="runtime">
<h1>Runtime<a class="headerlink" href="#runtime" title="Permalink to this heading"></a></h1>
<section id="buffermanager-h">
<h2>bufferManager.h<a class="headerlink" href="#buffermanager-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv412tensorrt_llm">
<span id="_CPPv312tensorrt_llm"></span><span id="_CPPv212tensorrt_llm"></span><span id="tensorrt_llm"></span><span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><a class="headerlink" href="#_CPPv412tensorrt_llm" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimeE">
<span id="_CPPv3N12tensorrt_llm7runtimeE"></span><span id="_CPPv2N12tensorrt_llm7runtimeE"></span><span id="tensorrt_llm::runtime"></span><span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimeE" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManagerE"></span><span id="tensorrt_llm::runtime::BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;bufferManager.h&gt;</em></div>
<p>A helper class for managing memory on host and device. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10IBufferPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10IBufferPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0bd5b422b1e3a5d2171f66532f27f413"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IBufferPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10ITensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10ITensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a5f8cccf8f635cf3cfbec353e1a7317b9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ITensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ae977afd5a423417ac57f3bffa8162b33"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::BufferManager::BufferManager__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ac641ea2cdcc436ed29ce3bacc42d549a"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferManager</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13BufferManagerE13CudaStreamPtr" title="Permalink to this definition"></a><br /></dt>
<dd><p>Construct a <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1BufferManager"><span class="std std-ref">BufferManager</span></a>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>cudaStream</strong> – <strong>[in]</strong> The cuda stream to use for all operations on GPU (allocation, de-allocation, copying, etc.). </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::gpu__std::s.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a2cf0e00fd6ac0cfe8b593749583a7a03"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuENSt6size_tEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the GPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::gpu__nvinfer1::Dims.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a5a023b2d5f629bc86e262f9733a12a59"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager3gpuEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the GPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::allocate__MemoryType.std::s.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1bdafec3fa633fdb0d321f5e748b3174"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeNSt6size_tEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size and memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::allocate__MemoryType.nvinfer1::Dims.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0e4e4c0e248e95e19061a61fadec30cc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8allocateE10MemoryTypeN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions and memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::emptyBuffer__MemoryType.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a0e78e3aa2aa1d635ae230996b449f9a4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyBuffer</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyBufferE10MemoryTypeN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Create an empty <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given memory type. It may be resized later. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::emptyTensor__MemoryType.nvinfer1::DataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1d0157fa19df03ba561d15d5b64de259"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyTensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager11emptyTensorE10MemoryTypeN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Create an empty <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given memory type. It may be reshaped later. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::setZero__IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1afe66c31bfc8e00ebbdb8405194be0db9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setZero</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager7setZeroER7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Set the contents of the given <code class="docutils literal notranslate"><span class="pre">buffer</span></code> to zero. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copy__voidCP.IBufferR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a1adc12c1388e8a3392ac39a3ae64d518"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">srcType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.voidP.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a313f2bd39b26e568c97f092a98ac20f9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dst</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dstType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::copy__voidCP.IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3086524a384108c45199076f536d4d9c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyEPKvR7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.voidPC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a583c0a02dcd51050b5b4273a19c0bde7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferPv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer"></span><span id="tensorrt_llm::runtime::BufferManager::copy__IBufferCR.IBufferRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a406d3f69a661d5a3bcb5dded51626c44"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copy</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dst</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager4copyERK7IBufferR7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> to <code class="docutils literal notranslate"><span class="pre">dst</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copyFrom__IBufferCR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3101d71e18fa84154b687236487dfc5f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7IBuffer10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType"></span><span id="tensorrt_llm::runtime::BufferManager::copyFrom__ITensorCR.MemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a80a336bdfbd0faf2d76e470354bcddf0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager8copyFromERK7ITensor10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a51f44f1cf97f9c0b686048d37291ea5a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10IBufferPtrRKNSt6vectorI1TEE10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromEP1TN8nvinfer14DimsE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromEP1TN8nvinfer14DimsE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1ac42305510a739c47d3c4358b97560f3e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrP1TN8nvinfer14DimsE10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType">
<span id="_CPPv3I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType"></span><span id="_CPPv2I0ENK12tensorrt_llm7runtime13BufferManager8copyFromERKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a8feac5727a72ee0d1c50b3ff5ee4e979"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyFrom</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType" title="tensorrt_llm::runtime::BufferManager::copyFrom::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">src</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4I0ENK12tensorrt_llm7runtime13BufferManager8copyFromE10ITensorPtrRKNSt6vectorI1TEEN8nvinfer14DimsE10MemoryType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Copy <code class="docutils literal notranslate"><span class="pre">src</span></code> into a new <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> with a potentially different memory type. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13BufferManager9getStreamEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13BufferManager9getStreamEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13BufferManager9getStreamEv"></span><span id="tensorrt_llm::runtime::BufferManager::getStreamC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aa8eb6eafae45f681f082cab5daa26029"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getStream</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13BufferManager9getStreamEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Get the underlying cuda stream. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::cpu__std::s.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a7edb3cdde9b85c01fec18f3f4f34282f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuENSt6size_tEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::cpu__nvinfer1::Dims.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3beb40d9ddabd96774167df148efc676"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager3cpuEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::pinned__std::s.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a6a58588cfc27c05c1d7dbb2f043672d3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10IBufferPtrE" title="tensorrt_llm::runtime::BufferManager::IBufferPtr"><span class="n"><span class="pre">IBufferPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedENSt6size_tEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates a pinned <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> of the given size on the CPU. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::BufferManager::pinned__nvinfer1::Dims.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1a3cfc4376c5e62876586e1e07e08d3069"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10ITensorPtrE" title="tensorrt_llm::runtime::BufferManager::ITensorPtr"><span class="n"><span class="pre">ITensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE"><span class="n"><span class="pre">kBYTE_TYPE</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager6pinnedEN8nvinfer14DimsEN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Allocates a pinned <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> of the given dimensions on the CPU. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE"></span><span id="tensorrt_llm::runtime::BufferManager::kBYTE_TYPE__auto"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1af4254e3a9525a8051face3c7ca729b65"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBYTE_TYPE</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager10kBYTE_TYPEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager7mStreamE"></span><span id="tensorrt_llm::runtime::BufferManager::mStream__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aeab688938251784e88b66cff01a0dde3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager7mStreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi">
<span id="_CPPv3N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi"></span><span id="_CPPv2N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi"></span><span id="tensorrt_llm::runtime::BufferManager::initMemoryPool__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferManager_1aba8fe4df93bbf829d0cefa8bcc63e657"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">initMemoryPool</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager14initMemoryPoolEi" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="common-h">
<h2>common.h<a class="headerlink" href="#common-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime8SizeTypeE"></span><span class="target" id="common_8h_1a8da5a8de61b5f096333bdd15d5a16875"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime11TokenIdTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TokenIdTypeE"></span><span class="target" id="common_8h_1a1de916f1c3f3c8d1f9c66320afc6df17"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenIdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime12StringPtrMapE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime12StringPtrMapE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="common_8h_1a43946c471b82feb36a6350de9cde277d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StringPtrMap</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unordered_map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE" title="tensorrt_llm::runtime::StringPtrMap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime12StringPtrMapE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</section>
<section id="cudaevent-h">
<h2>cudaEvent.h<a class="headerlink" href="#cudaevent-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEventE"></span><span id="tensorrt_llm::runtime::CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a765323cdb24844d31c8f9e354a5194f8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaEvent_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventEj">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent9CudaEventEj"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent9CudaEventEj"></span><span id="tensorrt_llm::runtime::CudaEvent::CudaEvent__unsigned-i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1aae9d65fd3cc3d42763c1219710dcd6ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">flags</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaEventDisableTiming</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventEj" title="Permalink to this definition"></a><br /></dt>
<dd><p>Creates a new cuda event. The event will be destroyed in the destructor.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>flags</strong> – Flags for event creation. By default, event timing is disabled. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb"></span><span id="tensorrt_llm::runtime::CudaEvent::CudaEvent__pointer.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a8ac4cb3f5ac924e72862c1c5fd033cbd"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaEvent</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsEvent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent9CudaEventE7pointerb" title="Permalink to this definition"></a><br /></dt>
<dd><p>Pass an existing cuda event to this object.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>event</strong> – The event to pass to this object. </p></li>
<li><p><strong>ownsEvent</strong> – Whether this object owns the event and destroys it in the destructor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent3getEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent3getEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent3getEv"></span><span id="tensorrt_llm::runtime::CudaEvent::getC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a530c756fe9e6ad149b813659b3644f16"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent3getEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the event associated with this object. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv"></span><span id="tensorrt_llm::runtime::CudaEvent::synchronizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a9216cdfafea99849fb47b1be60fb1a7e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">synchronize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent11synchronizeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Synchronizes the event. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent12element_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent12element_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a411326cde2f9f947c5d92abe724ebb34"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">element_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_pointer_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent8EventPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent8EventPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1ac5468fc310bab51276ed6d6212e7240f"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">EventPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent12element_typeE" title="tensorrt_llm::runtime::CudaEvent::element_type"><span class="n"><span class="pre">element_type</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE" title="tensorrt_llm::runtime::CudaEvent::Deleter"><span class="n"><span class="pre">Deleter</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent6mEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent6mEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent6mEventE"></span><span id="tensorrt_llm::runtime::CudaEvent::mEvent__EventPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1a3231de6317de1857e5f699a94d59dcf2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent8EventPtrE" title="tensorrt_llm::runtime::CudaEvent::EventPtr"><span class="n"><span class="pre">EventPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent6mEventE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7DeleterE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7DeleterE"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7DeleterE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::Deleter__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a7807c6a0240c9d01f09dbde19b422d05"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsEvent</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1aec3a7bdb695dc6598d6c6b8c5cbd4598"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter7DeleterEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer">
<span id="_CPPv3NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer"></span><span id="_CPPv2NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::call-operator__pointerCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a82af09350919fceb1f0f4c43d1822d37"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">()</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9CudaEvent7DeleterclE7pointer" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE">
<span id="_CPPv3N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE"></span><span id="tensorrt_llm::runtime::CudaEvent::Deleter::mOwnsEvent__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaEvent_1_1Deleter_1a4df7640a97930eaf43a1ca952f3f919f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOwnsEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7Deleter10mOwnsEventE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="cudastream-h">
<h2>cudaStream.h<a class="headerlink" href="#cudastream-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStreamE"></span><span id="tensorrt_llm::runtime::CudaStream"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamEji">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream10CudaStreamEji"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream10CudaStreamEji"></span><span id="tensorrt_llm::runtime::CudaStream::CudaStream__unsigned-i.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a2ba13a61587813c68c018a64ed2967fe"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">flags</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaStreamNonBlocking</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamEji" title="Permalink to this definition"></a><br /></dt>
<dd><p>Creates a new cuda stream on the current device. The stream will be destroyed in the destructor.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>flags</strong> – Flags for stream creation. See ::cudaStreamCreateWithFlags for a list of valid flags that can be passed. </p></li>
<li><p><strong>priority</strong> – Priority of the stream. Lower numbers represent higher priorities. See ::cudaDeviceGetStreamPriorityRange for more information about the meaningful stream priorities that can be passed. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib"></span><span id="tensorrt_llm::runtime::CudaStream::CudaStream__cudaStream_t.i.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a4ff09ea22fc6679e2d93b772e148d19e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStream</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">device</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsStream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream10CudaStreamE12cudaStream_tib" title="Permalink to this definition"></a><br /></dt>
<dd><p>Pass an existing cuda stream to this object.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>stream</strong> – The stream to pass to this object. </p></li>
<li><p><strong>device</strong> – The device on which the stream was created. </p></li>
<li><p><strong>ownsStream</strong> – Whether this object owns the stream and destroys it in the destructor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream9getDeviceEv"></span><span id="tensorrt_llm::runtime::CudaStream::getDeviceC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a2209ae12d3b5a27740d66bec35e686c9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream9getDeviceEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the device on which the stream was created. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream3getEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream3getEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream3getEv"></span><span id="tensorrt_llm::runtime::CudaStream::getC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a9ba739eedb71b18bc4a58ff480c82a27"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream3getEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the stream associated with this object. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream11synchronizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream11synchronizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream11synchronizeEv"></span><span id="tensorrt_llm::runtime::CudaStream::synchronizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a3e3def9cc0e09e9724e9e68232ed2679"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">synchronize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream11synchronizeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Synchronizes the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE"></span><span id="tensorrt_llm::runtime::CudaStream::record__CudaEvent::pointerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1acc03555968e2361dc08d28f2228a1e31"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">record</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordEN9CudaEvent7pointerE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Record an event on the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent"></span><span id="tensorrt_llm::runtime::CudaStream::record__CudaEventCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1ace6075f1266bdefe0b2033717dd8b14a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">record</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream6recordERK9CudaEvent" title="Permalink to this definition"></a><br /></dt>
<dd><p>Record an event on the stream. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE"></span><span id="tensorrt_llm::runtime::CudaStream::wait__CudaEvent::pointerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a97622ff55b119c5eec1b096115462098"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wait</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEvent7pointerE" title="tensorrt_llm::runtime::CudaEvent::pointer"><span class="n"><span class="pre">pointer</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitEN9CudaEvent7pointerE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for an event. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent"></span><span id="tensorrt_llm::runtime::CudaStream::wait__CudaEventCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a60cfbc942ee8d5f787ae66f891d91766"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wait</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream4waitERK9CudaEvent" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for an event. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream9StreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream9StreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a42e6eb53f179659c51fbd8981383af64"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_pointer_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">cudaStream_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE" title="tensorrt_llm::runtime::CudaStream::Deleter"><span class="n"><span class="pre">Deleter</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7mStreamE"></span><span id="tensorrt_llm::runtime::CudaStream::mStream__StreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a21588e234ee724b69f92d1ef26779334"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream9StreamPtrE" title="tensorrt_llm::runtime::CudaStream::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mStreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7mDeviceE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7mDeviceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7mDeviceE"></span><span id="tensorrt_llm::runtime::CudaStream::mDevice__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1a5dc4aea4b9abfa576f166ce152b675e0"></span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDevice</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7mDeviceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7DeleterE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7DeleterE"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7DeleterE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::Deleter__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1aa03f43a2e696a0433dd1a1c14ce129c2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">ownsStream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::Deleter"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1adae80ccd16ccabd753f118cdb4111a51"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Deleter</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter7DeleterEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t">
<span id="_CPPv3NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t"></span><span id="_CPPv2NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::call-operator__cudaStream_tCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1a077db80f4a8d30a5f2e0adde1ec6372a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">()</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaStream_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10CudaStream7DeleterclE12cudaStream_t" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE"></span><span id="tensorrt_llm::runtime::CudaStream::Deleter::mOwnsStream__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1CudaStream_1_1Deleter_1a71b80a1570697e35d7b5edb51904d9bd"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOwnsStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10CudaStream7Deleter11mOwnsStreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="decodinginput-h">
<h2>decodingInput.h<a class="headerlink" href="#decodinginput-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInputE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInputE"></span><span id="tensorrt_llm::runtime::DecodingInput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a8e97b9538b8259558b0c15cbe2a59d15"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType9TensorPtr9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType9TensorPtr9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType9TensorPtr9TensorPtr"></span><span id="tensorrt_llm::runtime::DecodingInput::DecodingInput__SizeType.SizeType.TensorPtr.TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a4dc28cfb590912072e86070136f4e40a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13DecodingInputE8SizeType8SizeType9TensorPtr9TensorPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput4stepE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput4stepE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput4stepE"></span><span id="tensorrt_llm::runtime::DecodingInput::step__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a82755273657729ca89f9186abeeb713c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">step</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput4stepE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9maxLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9maxLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9maxLengthE"></span><span id="tensorrt_llm::runtime::DecodingInput::maxLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1ac4e6e64f43c3994f785a7047afd4578e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9maxLengthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput9batchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput9batchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput9batchSizeE"></span><span id="tensorrt_llm::runtime::DecodingInput::batchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a5d6979ade3dcff577fcf46c70c887917"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">batchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9batchSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput6logitsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput6logitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput6logitsE"></span><span id="tensorrt_llm::runtime::DecodingInput::logits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1aafadc431e8d06615d8425d82b1afaaa8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6logitsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput6endIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput6endIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput6endIdsE"></span><span id="tensorrt_llm::runtime::DecodingInput::endIds__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6bb630897ccbb4f437cf9ec517912d25"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput6endIdsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE"></span><span id="tensorrt_llm::runtime::DecodingInput::sequenceLimitLength__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1aa14e54120b58626beca8705ced84a79c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLimitLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput19sequenceLimitLengthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE"></span><span id="tensorrt_llm::runtime::DecodingInput::embeddingBias__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a132486f973befa6c1c54bb0c22aab289"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBias</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13embeddingBiasE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput7lengthsE"></span><span id="tensorrt_llm::runtime::DecodingInput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1ad44428022172d330f631c81a712ce206"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput7lengthsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput12badWordsListE"></span><span id="tensorrt_llm::runtime::DecodingInput::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1acfc2ebcd8202fc2330466a8773b1ab6d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput12badWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput13stopWordsListE"></span><span id="tensorrt_llm::runtime::DecodingInput::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6979dbc3a119fefd9f50c2132c1c95e7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput13stopWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE"></span><span id="tensorrt_llm::runtime::DecodingInput::noRepeatNgramSize__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a764f889cfac79873d7c88978e9d42df4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">noRepeatNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput17noRepeatNgramSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::DecodingInput::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingInput_1a6eccbda69f8c35e1eda57e9eb24ca930"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput9TensorPtrE" title="tensorrt_llm::runtime::DecodingInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInput16cacheIndirectionE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="decodingoutput-h">
<h2>decodingOutput.h<a class="headerlink" href="#decodingoutput-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutputE"></span><span id="tensorrt_llm::runtime::DecodingOutput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1aeb215aae60278c44bcdbd17ae5f7c8e1"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr"></span><span id="tensorrt_llm::runtime::DecodingOutput::DecodingOutput__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a33bcc26e94af9077c4c8d1ba65acd39c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14DecodingOutputE9TensorPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput3idsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a9937790aaf8c5cbb5230236afc7656be"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput3idsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9newTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9newTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9newTokensE"></span><span id="tensorrt_llm::runtime::DecodingOutput::newTokens__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a1c92d23772ed90b78f9ac8c86f46d9fd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9newTokensE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput8finishedE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput8finishedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput8finishedE"></span><span id="tensorrt_llm::runtime::DecodingOutput::finished__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a721bfb18d093d881f2f3a7d288dfb0e4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finished</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8finishedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput11finishedSumE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput11finishedSumE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput11finishedSumE"></span><span id="tensorrt_llm::runtime::DecodingOutput::finishedSum__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a2bb2c2cc930598e59363a8d61e2f0fc3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finishedSum</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11finishedSumE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput8logProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a4abf75dc398349212b15b93f843fc03c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput8logProbsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::cumLogProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a428f7d981f0617bda37e6f64bb4f0007"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput11cumLogProbsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput9parentIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput9parentIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput9parentIdsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::parentIds__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a9f518ec59bd0df527432074008d974a6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parentIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9parentIdsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput7lengthsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a7425f953e704cffb2b917d475fc90c12"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput7lengthsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::DecodingOutput::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a6ee6dfc4bf8f6f7ae17e0fc8d11ea56d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput16cacheIndirectionE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE"></span><span id="tensorrt_llm::runtime::DecodingOutput::beamHypotheses__BeamHypotheses"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1a1e1c48231792a45618deff974b48b4ee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"><span class="n"><span class="pre">BeamHypotheses</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamHypotheses</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14beamHypothesesE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE"></span><span id="tensorrt_llm::runtime::DecodingOutput::kNegativeInfinity__float"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1afad8e90f48ab5da093fdbb84b801f6da"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kNegativeInfinity</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1e20f</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput17kNegativeInfinityE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamHypotheses</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::empty__BufferManagerR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a00c82796e448818892e94f8a064a74cd"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">empty</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5emptyER13BufferManager" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::reshape__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a5ca4a3c3e34fe1f94224866821a2c352"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reshape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7reshapeE8SizeType8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::release"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a6c10ab322d436f29ce99f0f0cb4e1e1a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">release</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses7releaseEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::init__BufferManagerR.TokenIdType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1ab42b472a1d818532cd173ba69ad6097e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">init</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TokenIdTypeE" title="tensorrt_llm::runtime::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses4initER13BufferManager11TokenIdType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::slice__SizeType.SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a35faed30b9315b8e1890c64d77561ac1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypothesesE" title="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses"><span class="n"><span class="pre">BeamHypotheses</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIndex</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses5sliceE8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::outputIdsTgt__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1aaeebc4faabecf671bd786c5ff44e2364"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">outputIdsTgt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12outputIdsTgtE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::sequenceLengthsTgt__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a42ed879ae2b5475b54aa7d3328b5357b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLengthsTgt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses18sequenceLengthsTgtE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::cumLogProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1ad2aaa38891d139affc6713f4d1a6e207"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses11cumLogProbsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::normedScores__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1af97b849bf039b95020a3e9d7601b77f0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">normedScores</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses12normedScoresE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a51b9823628bb1388d261bbd18b4e141e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8logProbsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::minNormedScores__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a545ce50c9d6ba29ebd84bbe1c7379648"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">minNormedScores</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses15minNormedScoresE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::numBeams__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a207a33747d2474bc9252b6105e1cf7e9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numBeams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses8numBeamsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE">
<span id="_CPPv3N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE"></span><span id="_CPPv2N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE"></span><span id="tensorrt_llm::runtime::DecodingOutput::BeamHypotheses::isDone__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1DecodingOutput_1_1BeamHypotheses_1a75ad5911c7aa09ed3214d0d2ee2e4bad"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput9TensorPtrE" title="tensorrt_llm::runtime::DecodingOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isDone</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutput14BeamHypotheses6isDoneE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="generationinput-h">
<h2>generationInput.h<a class="headerlink" href="#generationinput-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInputE"></span><span id="tensorrt_llm::runtime::GenerationInput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationInput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1ad4d928e8ccca5918cf0f158377a91334"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb"></span><span id="tensorrt_llm::runtime::GenerationInput::GenerationInput__SizeTypeC.SizeTypeC.TensorPtr.TensorPtr.b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1a919fd8de0e3905a059b746f1e9e35dbe"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationInput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">lengths</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">packed</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput15GenerationInputEK8SizeTypeK8SizeType9TensorPtr9TensorPtrb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput5endIdE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput5endIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput5endIdE"></span><span id="tensorrt_llm::runtime::GenerationInput::endId__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1a45567fc133775036062587d9ece61eb8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput5endIdE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput5padIdE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput5padIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput5padIdE"></span><span id="tensorrt_llm::runtime::GenerationInput::padId__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1adf789d32814e781d64e3abe485acc36a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">padId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput5padIdE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput3idsE"></span><span id="tensorrt_llm::runtime::GenerationInput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1aa8bd15f4c48f06d5270c4bd860b16c00"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput3idsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput7lengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput7lengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput7lengthsE"></span><span id="tensorrt_llm::runtime::GenerationInput::lengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1a6e9afbe2aabafc722c81144d258e88fa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput7lengthsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput6packedE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput6packedE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput6packedE"></span><span id="tensorrt_llm::runtime::GenerationInput::packed__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1ae16b0e7b06def770e9e0037532d2bfe2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">packed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput6packedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput16embeddingBiasOptE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput16embeddingBiasOptE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput16embeddingBiasOptE"></span><span id="tensorrt_llm::runtime::GenerationInput::embeddingBiasOpt__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1a0976efe9d64d9aeb7a6af8acadf32958"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBiasOpt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput16embeddingBiasOptE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput12badWordsListE"></span><span id="tensorrt_llm::runtime::GenerationInput::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1ae9a6430ce531bfc02be2ee95b1809e5b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput12badWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput13stopWordsListE"></span><span id="tensorrt_llm::runtime::GenerationInput::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1afa302004663b93df44247bedc31d1fa1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput9TensorPtrE" title="tensorrt_llm::runtime::GenerationInput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput13stopWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GenerationInput12maxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime15GenerationInput12maxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GenerationInput12maxNewTokensE"></span><span id="tensorrt_llm::runtime::GenerationInput::maxNewTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationInput_1aad080660c7e4cf593d37f62632dbf63b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInput12maxNewTokensE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="generationoutput-h">
<h2>generationOutput.h<a class="headerlink" href="#generationoutput-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutputE"></span><span id="tensorrt_llm::runtime::GenerationOutput"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1ac2bcdfc419f03bf441a1d863a5032aba"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput8CallbackE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput8CallbackE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput8CallbackE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a039ff4327d4017bde5e6ca805bae887e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Callback</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n"><span class="pre">ids</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n"><span class="pre">step</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n"><span class="pre">finished</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput8CallbackE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr"></span><span id="tensorrt_llm::runtime::GenerationOutput::GenerationOutput__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a36b18b313b4fe3aa0c765dd99b69fc64"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GenerationOutput</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput16GenerationOutputE9TensorPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput3idsE"></span><span id="tensorrt_llm::runtime::GenerationOutput::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1abe34ae6955cb9b92a3737c3ff8007d31"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput3idsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput8logProbsE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput8logProbsE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput8logProbsE"></span><span id="tensorrt_llm::runtime::GenerationOutput::logProbs__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1ad05894034e839afc1a5b136aea1ec5b7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput8logProbsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput13contextLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput13contextLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput13contextLogitsE"></span><span id="tensorrt_llm::runtime::GenerationOutput::contextLogits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a2f45fb2c7cf0ab1c4d93098afdf60fa3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput9TensorPtrE" title="tensorrt_llm::runtime::GenerationOutput::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput13contextLogitsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16GenerationOutput16onTokenGeneratedE">
<span id="_CPPv3N12tensorrt_llm7runtime16GenerationOutput16onTokenGeneratedE"></span><span id="_CPPv2N12tensorrt_llm7runtime16GenerationOutput16onTokenGeneratedE"></span><span id="tensorrt_llm::runtime::GenerationOutput::onTokenGenerated__Callback"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GenerationOutput_1a01e4b5ca66e302f380c156896849ee91"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput8CallbackE" title="tensorrt_llm::runtime::GenerationOutput::Callback"><span class="n"><span class="pre">Callback</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">onTokenGenerated</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutput16onTokenGeneratedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptdecoder-h">
<h2>gptDecoder.h<a class="headerlink" href="#gptdecoder-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm6layersE">
<span id="_CPPv3N12tensorrt_llm6layersE"></span><span id="_CPPv2N12tensorrt_llm6layersE"></span><span id="tensorrt_llm::layers"></span><span class="target" id="namespacetensorrt__llm_1_1layers"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">layers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm6layersE" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm6layers18DynamicDecodeLayerE">
<span id="_CPPv3I0EN12tensorrt_llm6layers18DynamicDecodeLayerE"></span><span id="_CPPv2I0EN12tensorrt_llm6layers18DynamicDecodeLayerE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1layers_1_1DynamicDecodeLayer"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DynamicDecodeLayer</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm6layers18DynamicDecodeLayerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10GptDecoderE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10GptDecoderE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoder</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a8b5c0f02247cc75b5d121681c1990d3b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::GptDecoder::GptDecoder__s.s.CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a0cbdae7bd365d6175f9fe78a0237cc7c"></span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoder</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoder::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10GptDecoderE6size_t6size_tRK13CudaStreamPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t"></span><span id="tensorrt_llm::runtime::GptDecoder::setup__SamplingConfigCR.s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1af426907b7aac4424847ee592c0596585"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder5setupERK14SamplingConfig6size_t" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::GptDecoder::forward__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a687dcb28ec2d720be9c35694250d346f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder7forwardER14DecodingOutputRK13DecodingInput" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::GptDecoder::forwardAsync__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1af707cd75a441ad7c639536c75459c65e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder8mManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder8mManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder8mManagerE"></span><span id="tensorrt_llm::runtime::GptDecoder::mManager__BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a8602244879ca7e9a83ccb7811cea26a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder8mManagerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder10mAllocatorE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder10mAllocatorE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder10mAllocatorE"></span><span id="tensorrt_llm::runtime::GptDecoder::mAllocator__common::CudaAllocator"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1a51093c91db9709a5cdb1d6004c19ae57"></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CudaAllocator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mAllocator</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder10mAllocatorE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE"></span><span id="tensorrt_llm::runtime::GptDecoder::mDynamicDecodeLayer__std::shared_ptr:tensorrt_llm::layers::DynamicDecodeLayer:T::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoder_1ae47acd96842e4ebc4f34af4fc4370746"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm6layersE" title="tensorrt_llm::layers"><span class="n"><span class="pre">layers</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm6layers18DynamicDecodeLayerE" title="tensorrt_llm::layers::DynamicDecodeLayer"><span class="n"><span class="pre">DynamicDecodeLayer</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10GptDecoderE" title="tensorrt_llm::runtime::GptDecoder::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDynamicDecodeLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptDecoder19mDynamicDecodeLayerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoderE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoderE"></span><span id="tensorrt_llm::runtime::IGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1GptDecoder"><span class="std std-ref">tensorrt_llm::runtime::GptDecoder&lt; T &gt;</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoderD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoderD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoderD0Ev"></span><span id="tensorrt_llm::runtime::IGptDecoder::~IGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a5ad01b4d1a088b47e56ce6bed969cb57"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~IGptDecoder</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderD0Ev" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t"></span><span id="tensorrt_llm::runtime::IGptDecoder::setup__SamplingConfigCR.s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a3dffd84af10329dbc69291086ac931e4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder5setupERK14SamplingConfig6size_t" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::IGptDecoder::forward__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a892c655936e25a38a7f0737fbfadfc1d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder7forwardER14DecodingOutputRK13DecodingInput" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput"></span><span id="tensorrt_llm::runtime::IGptDecoder::forwardAsync__DecodingOutputR.DecodingInputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1a8f87707fea456a6decd013b6b831d336"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder12forwardAsyncER14DecodingOutputRK13DecodingInput" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager"></span><span id="tensorrt_llm::runtime::IGptDecoder::gatherTree__ITensorR.DecodingOutputCR.DecodingInputCR.BufferManagerCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1ae754c28ed1ed7f31726ccbc6ebe385ab"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gatherTree</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">finalOutputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingOutput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingInput</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">manager</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder10gatherTreeER7ITensorRK14DecodingOutputRK13DecodingInputRK13BufferManager" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE"></span><span id="tensorrt_llm::runtime::IGptDecoder::create__nvinfer1::DataType.s.s.BufferManager::CudaStreamPtrCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoder_1af9c668b9e5cdd2e0a65bfd55ba9b8b98"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">create</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManager13CudaStreamPtrE" title="tensorrt_llm::runtime::BufferManager::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoder6createEN8nvinfer18DataTypeE6size_t6size_tRKN13BufferManager13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptdecoderbatch-h">
<h2>gptDecoderBatch.h<a class="headerlink" href="#gptdecoderbatch-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatchE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatchE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatchE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderBatch</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE" title="tensorrt_llm::runtime::IGptDecoderBatch"><span class="n"><span class="pre">IGptDecoderBatch</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatchE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;gptDecoderBatch.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a8206454db9d639c251910c03681a9ed0"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1afda009f36f02088447dc1da45e5e94f4"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::GptDecoderBatch__std::s.std::s.CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1af478d2ea92161498aefa5ddb70d36262"></span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderBatch</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSizePadded</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15GptDecoderBatchENSt6size_tENSt6size_tE13CudaStreamPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::setup__SizeType.SizeType.SizeType.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac5f1b0f5fb9ee4322b50ace3e7d22301"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Setup the decoder before calling <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ab7e07276f33bfa6ef3c90baa49ee6eec"><span class="std std-ref"><span class="pre">forward()</span></span></a></code></p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::newRequest__SizeType.decoder_batch::RequestCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7214c20bc16c253fd2cb199884cd9859"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="tensorrt_llm::runtime::decoder_batch::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd><p>Initialize the decoder at <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code> with a new <code class="docutils literal notranslate"><span class="pre">request</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::newBatch__GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4362ed782250a10e670f1db2b661cd0b"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8newBatchERK15GenerationInputRK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd><p>Initialize the decoder with new batch of inputs. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardAsync__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aa468001b519ad9767958cb8e660faea5"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">TokenPtr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host process and return the token for synchronization. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardSync__decoder_batch::TokenCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a495a39bf71a7d39cd81495834716760f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">e</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for the call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> associated with a token to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::forwardAsync__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab0220317a75f65030f59878e26e3d737"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12forwardAsyncERN7decoder6OutputERKN7decoder5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host thread. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14isFinishedSyncEv">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch14isFinishedSyncEv"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch14isFinishedSyncEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::isFinishedSync"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ac9935f86503e3857f8d2563532531e09"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFinishedSync</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14isFinishedSyncEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for the last call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> to complete and return whether all sequences have finished. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4c69dab125603dd2a4fbd9ab043afa30"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch11getFinishedEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize], indicators of finished requests </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a313a9ae3a464f84398b9ba038b4af554"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a445fc8f3f0db689b2166cb15fdc63df2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getOutputIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getFinalOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab3bb06c1c21e53f4597a4a4e40605539"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinalOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Execute postProcessRequest and returns OutputIds for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>. Result will only be available after event returned </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getFinalOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a222f485382bc2c429686df944c3556f0"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinalOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch17getFinalOutputIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Execute postProcessRequest and returns OutputIds. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getParentIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a93178ec5242faf2de970d4d017352dae"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParentIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getParentIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains parent ids collected during beam search without padding, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch16getFinishedBeamsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch16getFinishedBeamsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch16getFinishedBeamsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getFinishedBeamsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9fd74a125803979c32a246d8bab2d188"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinishedBeams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch16getFinishedBeamsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], marks finished requests (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch16getOutputLengthsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch16getOutputLengthsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch16getOutputLengthsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getOutputLengthsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab1b79b788d470a0a295ac4e940ac2338"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputLengths</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch16getOutputLengthsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], total sequence lengths (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getCumLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a3c07c5204d41a4dfdf688a4482d510bb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch14getCumLogProbsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a0f899f3a1a1101cf24375fbd68eff1a9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch12getNewTokensEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, maxBeamWidth], tokens generated in last forward pass, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNbStepsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9c9a6a07562637a290ed93e987bd06a7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbSteps</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch10getNbStepsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize], the number of generation steps executed on each request </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::getNbFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7c4d967f7646e5a78f50d4ac6844be93"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch13getNbFinishedEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[1], number of finished sequences, in pinned host memory </p>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1adbc3e2ab6fcff2a86fee1974fd2ea358"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptDecoderPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11IGptDecoderE" title="tensorrt_llm::runtime::IGptDecoder"><span class="n"><span class="pre">IGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aebed7ff7ea091a472908a7d2cc468684"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingInputPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13DecodingInputE" title="tensorrt_llm::runtime::DecodingInput"><span class="n"><span class="pre">DecodingInput</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a8b297105bd88a4066694adf8a91d4073"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingOutputPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14DecodingOutputE" title="tensorrt_llm::runtime::DecodingOutput"><span class="n"><span class="pre">DecodingOutput</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::postProcessRequest__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7c4def0f8b1239b53724fbf6cf576b12"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">postProcessRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime15GptDecoderBatch18postProcessRequestE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Gather final results for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code></p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mVocabSize__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a91f2563dd7fecc08ba0da2fa073d7763"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch10mVocabSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mVocabSizePadded__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1af9abcb9a1802cfe924e5b7ac0e03b98c"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSizePadded</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mVocabSizePaddedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mStream__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a33675102b6406b35993f06c59508b4fd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch7mStreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mBufferManager__BufferManager"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a73a99bf969ac6a13872b1b93f443bc2d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBufferManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch14mBufferManagerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mForwardToken__TokenPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1abe39442ac571c5827d98ec295bd0513c"></span><span class="n"><span class="pre">TokenPtr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mForwardToken</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardTokenE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mForwardEvent__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a9d42242aa2aac2bffd0387a6272f57f9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mForwardEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mForwardEventE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mStreams__std::vector:CudaStreamPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a5b1a9b00ef30809bef69336e5adfcc6c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13CudaStreamPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStreams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mStreamsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecoders__std::vector:GptDecoderPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a656cea9abf1ea0c8b3697a79ab51dde9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13GptDecoderPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::GptDecoderPtr"><span class="n"><span class="pre">GptDecoderPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoders</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mDecodersE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecodingInputs__std::vector:DecodingInputPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1aeb33339a3b75a2152134fdacd2660786"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingInputPtr"><span class="n"><span class="pre">DecodingInputPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingInputs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch15mDecodingInputsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mDecodingOutputs__std::vector:DecodingOutputPtr:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab1c5ab4afe168288cb33331a486d5be8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingOutputPtr"><span class="n"><span class="pre">DecodingOutputPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingOutputs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mDecodingOutputsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingInput__DecodingInputPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a73642bc20fd9bcd68b262edd55d4d2a7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16DecodingInputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingInputPtr"><span class="n"><span class="pre">DecodingInputPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mJointDecodingInput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch19mJointDecodingInputE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mJointDecodingOutput__DecodingOutputPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a09890f57577749180eb23ba792989e96"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch17DecodingOutputPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::DecodingOutputPtr"><span class="n"><span class="pre">DecodingOutputPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mJointDecodingOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch20mJointDecodingOutputE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mNbSteps__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1ab82d738a6ee4eb24160c4203f50ba47b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbSteps</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch8mNbStepsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mFinished__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a4bacfb74153463dd2ea29e9e58d62139"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFinished</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9mFinishedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mFinishedSum__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a5e29dd7a16e6dcfdbc6af1f5a30d454d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::GptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFinishedSum</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch12mFinishedSumE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxNewTokens__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1adda26d8626cb810bd54f0eae099f81d6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch13mMaxNewTokensE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mBeamWidths__std::vector:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a62af2790978fdbdce912bb55d1538c7a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamWidths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch11mBeamWidthsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mMaxSequenceLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a17b9d45dd6c74bddca33b2edb555fe3c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxSequenceLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch18mMaxSequenceLengthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE"></span><span id="tensorrt_llm::runtime::GptDecoderBatch::mActualBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptDecoderBatch_1a7e2f18c7b2ebe3d76b568c46380e92c4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mActualBatchSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime15GptDecoderBatch16mActualBatchSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptjsonconfig-h">
<h2>gptJsonConfig.h<a class="headerlink" href="#gptjsonconfig-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfigE"></span><span id="tensorrt_llm::runtime::GptJsonConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptJsonConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig"></span><span id="tensorrt_llm::runtime::GptJsonConfig::GptJsonConfig__ss.ss.SizeType.SizeType.GptModelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a66216254b0b7a635b772546973578fd8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptJsonConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">name</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig13GptJsonConfigENSt6stringENSt6stringE8SizeType8SizeTypeRK14GptModelConfig" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a124fb0206f4ac3ff47629bcb8147589d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getModelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14getModelConfigEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getNameC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a929bf268a1dd42fa06b6d05834bd6646"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig7getNameEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getPrecisionC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ab5c3f3e5854ebdbd1b0f0d26fa8c2a6f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getPrecision</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getPrecisionEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getTensorParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1af3d5d313552e8497406ce3ae6b18bc11"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig20getTensorParallelismEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getPipelineParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a2ec48773fe7b568d771f53770d9b6b0d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig22getPipelineParallelismEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv"></span><span id="tensorrt_llm::runtime::GptJsonConfig::getWorldSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a589461cb3bd53963212f1eda119d7f2e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWorldSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig12getWorldSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::engineFilename__WorldConfigCR.ssCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ac663861699ef7f573f64d5fdb89f14af"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">engineFilename</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">model</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfigRKNSt6stringE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig">
<span id="_CPPv3NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig"></span><span id="_CPPv2NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig"></span><span id="tensorrt_llm::runtime::GptJsonConfig::engineFilename__WorldConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a9ce08952a809fbe5859a685215dd7258"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">engineFilename</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime13GptJsonConfig14engineFilenameERK11WorldConfig" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__ssCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a258c808e0e2c5445c807ae2f0a257782"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">json</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt6stringE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__isR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1af74ea4db9854b1f0ddd8aaa0d4a48f39"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">json</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERNSt7istreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::parse__std::filesystem::pathCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a959c20c852124ced898b12bbdeb5d79f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfigE" title="tensorrt_llm::runtime::GptJsonConfig"><span class="n"><span class="pre">GptJsonConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">path</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5parseERKNSt10filesystem4pathE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5mNameE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig5mNameE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig5mNameE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mName__ssC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a63163fef3a6b49203c7cf6cbb0855971"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mName</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig5mNameE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mPrecision__ssC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1ad17448c5f6c59a7832c433ceec3687ff"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPrecision</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig10mPrecisionE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mTensorParallelism__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a8f0596343a2ad81565cea45b5ac7f6fc"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensorParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig18mTensorParallelismE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mPipelineParallelism__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a69e73e4c48faf1f76ebfc16bcb544431"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig20mPipelineParallelismE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE"></span><span id="tensorrt_llm::runtime::GptJsonConfig::mGptModelConfig__GptModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptJsonConfig_1a573b497d8cd2a5197db8544a102c7af0"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGptModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13GptJsonConfig15mGptModelConfigE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptmodelconfig-h">
<h2>gptModelConfig.h<a class="headerlink" href="#gptmodelconfig-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfigE"></span><span id="tensorrt_llm::runtime::GptModelConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ModelVariant</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="Permalink to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635aeb854c1219d60212d16ad06f0d454dc8"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGpt</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGptE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5ab7e0ffcb31d8d6d19afd863370a635a30a41043cc5346b26f02b1a3e1cf54d5"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGlm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariant4kGlmE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::GptModelConfig__SizeType.SizeType.SizeType.SizeType.nvinfer1::DataTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a93a4b8bc5df2e1a25d4b876fdefb1ff6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptModelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">vocabSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbLayers</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbHeads</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">hiddenSize</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14GptModelConfigE8SizeType8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getVocabSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aff992ef62ed1999326b48800913b2bee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVocabSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getVocabSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::getVocabSizePadded__SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1acf1d9e9818c254ee026c012388025480"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVocabSizePadded</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18getVocabSizePaddedE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbLayers__SizeTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a527a93c79413842c6706295232b0b9ed"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbLayers</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getNbLayersE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbHeadsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a00ef36e3eb9bd188a911b7f80c345e0f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbHeads</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig10getNbHeadsEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getNbKvHeadsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a51a65e744ce4675cbdaaff83f16f2c54"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbKvHeads</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getNbKvHeadsEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setNbKvHeads__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ad837f99f5aafdcc1ef7baa15c86617b5"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNbKvHeads</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nbKvHeads</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setNbKvHeadsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getHiddenSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9df3d4f4da4ad6c30e6c91419d6b268f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHiddenSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig13getHiddenSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getSizePerHeadCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a70409542e05a302a44ac2d571aea25db"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizePerHead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getSizePerHeadEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getDataTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab0dd6dd6a8b7cacfe4f60716066f8613"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig11getDataTypeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::useGptAttentionPluginCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2eedc76ee7dc4712de143bcdc02f40f9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useGptAttentionPlugin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::useGptAttentionPlugin__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ac3723398406d29b02be8a06912ee0d82"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useGptAttentionPlugin</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useGptAttentionPlugin</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21useGptAttentionPluginEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePackedInputCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a4894eb60ae226002fda29841977f7935"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePackedInput</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14usePackedInputEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePackedInput__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1af71f4a0b8a8a8fc6e21ea28612b81ed1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePackedInput</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">inputPacked</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14usePackedInputEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePagedKvCacheCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a173a85af1a71420f692612ca00c87d69"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePagedKvCache</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::usePagedKvCache__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a80165e232642897d7c4fb621bb61fed3"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePagedKvCache</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pagedKvCache</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15usePagedKvCacheEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getTokensPerBlockCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a0fa158a8e4b14102c6ee79a717d40879"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokensPerBlock</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig17getTokensPerBlockEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setTokensPerBlock__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a443a5a8b94f03dd8b0a8fbbec298d3f8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTokensPerBlock</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">TokensPerBlock</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig17setTokensPerBlockE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getQuantModeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa35c989f165bd8aa9437c3c6ba7d0a75"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getQuantMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig12getQuantModeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::setQuantMode__common::QuantModeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a41fe2aa1fdd76ce76301859981425417"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setQuantMode</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">QuantMode</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12setQuantModeEN6common9QuantModeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::supportsInflightBatchingCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a275c4804ce2373f0c977c7a5f2ddd2a0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">supportsInflightBatching</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig24supportsInflightBatchingEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxBatchSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a24b66ade6d982663d81fa7b42d9c6055"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBatchSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxBatchSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxBatchSize__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab784bd1a00eb729f402abe8875b564ea"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBatchSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxBatchSizeE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxInputLenCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1af1e370c6c5bbf9dfc854e12c14cccb79"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxInputLen</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig14getMaxInputLenEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxInputLen__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab80523799abed725a58348c4d56c8ad8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxInputLen</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxInputLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig14setMaxInputLenE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxOutputLenCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aaa48e78ace7bb49aa634f3cb84ed7ac4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxOutputLen</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxOutputLenEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxOutputLen__SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab5e91bd0f16db6355ab0e0a14378658a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxOutputLen</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxOutputLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxOutputLenE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getMaxNumTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5065aee21fa4e99f5c52bad69117a5cb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getMaxNumTokensEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::GptModelConfig::setMaxNumTokens__std::optional:SizeType:CE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a06b325b04c1b396a0430320435f81373"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setMaxNumTokensENSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeContextLogitsCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2f0e4d2c822ec1a83c858674ffce5ec2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeContextLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::computeContextLogits__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a33188908109cfcc7baa824e284a1af8e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">computeContextLogits</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">computeContextLogits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig20computeContextLogitsEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::getModelVariantC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9e832f202599100dfac6af143f8c72b4"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getModelVariant</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig15getModelVariantEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant"></span><span id="tensorrt_llm::runtime::GptModelConfig::setModelVariant__ModelVariant"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a5fd810a5a863b75656d6c0914b2d5762"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setModelVariant</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelVariant</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15setModelVariantE12ModelVariant" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv"></span><span id="tensorrt_llm::runtime::GptModelConfig::useCustomAllReduceCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a392c5d57f071a1a36fa628580975faf9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useCustomAllReduce</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb"></span><span id="tensorrt_llm::runtime::GptModelConfig::useCustomAllReduce__bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a79e7448c925f5d6785575250d7a1971f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useCustomAllReduce</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">customAllReduce</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig18useCustomAllReduceEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mVocabSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a18e1092a5c44a311577bc4954efc9ad8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVocabSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mVocabSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbLayers__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a83dd559a542a2ebfe932387d4bc65faa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbLayers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mNbLayersE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbHeads__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a3aed4484b10e66d10c0f2e330aea1f06"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbHeads</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig8mNbHeadsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mNbKvHeads__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aafe290665c7ed003ec2347f04f1a54a5"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNbKvHeads</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mNbKvHeadsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mHiddenSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a43e64ac4881c3a9eee58b7dee25964e2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHiddenSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig11mHiddenSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mDataType__nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a497a3a345c3e48362790139c6fe56526"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig9mDataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mUseGptAttentionPlugin__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2c837cd8fccbcae0427ccb0334ddd9d2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUseGptAttentionPlugin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig22mUseGptAttentionPluginE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mInputPacked__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a780f8ca1c1f7565694104b0fa5bf45fb"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInputPacked</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mInputPackedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mPagedKvCache__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2b70f372d6c64c92f1ad003043dcdb52"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPagedKvCache</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mPagedKvCacheE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mTokensPerBlock__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa5ad1102f45507190c50ccfbd74f0788"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokensPerBlock</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig15mTokensPerBlockE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mQuantMode__common::QuantMode"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a9a5da6ac7818f7df4f68e3306653598b"></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">QuantMode</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mQuantMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig10mQuantModeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxBatchSize__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a8714ccf704b0a20be40bb99e229c2439"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxBatchSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxInputLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ab4be5cfe878d1333c6add3b824b6a7e4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxInputLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12mMaxInputLenE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxOutputLen__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1adc7f72184ca681976df8197f475942c2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxOutputLen</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxOutputLenE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mMaxNumTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1aa20bf7021d240c626fa6210511c31c47"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNumTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mMaxNumTokensE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mComputeContextLogits__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a49736d7c35c69ae8eb50892e69dcfeb3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mComputeContextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig21mComputeContextLogitsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mModelVariant__ModelVariant"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1ae45c6e3f4a0ae12a5df7ba02281b544e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig12ModelVariantE" title="tensorrt_llm::runtime::GptModelConfig::ModelVariant"><span class="n"><span class="pre">ModelVariant</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mModelVariant</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig13mModelVariantE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE">
<span id="_CPPv3N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE"></span><span id="_CPPv2N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE"></span><span id="tensorrt_llm::runtime::GptModelConfig::mUseCustomAllReduce__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptModelConfig_1a2475ff83565a03603f3a3e61425320d6"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUseCustomAllReduce</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfig19mUseCustomAllReduceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="gptsession-h">
<h2>gptSession.h<a class="headerlink" href="#gptsession-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_managerE"></span><span id="tensorrt_llm::batch_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">batch_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="tensorrt_llm::batch_manager::kv_cache_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager_1_1kv__cache__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kv_cache_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSessionE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSessionE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSessionE"></span><span id="tensorrt_llm::runtime::GptSession"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSessionE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9LoggerPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9LoggerPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1abf601de29903bb11af9d3d2976ea08a6"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoggerPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__GptModelConfigCR.WorldConfigCR.voidCP.std::s.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a4b3fd438443e516007a6beb6a27f72dc"></span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">engineSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigPKvNSt6size_tE9LoggerPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__GptModelConfigCR.WorldConfigCR.std::vector:uint8_t:CR.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a46e03f47ac584b26e20a6429cdacdf18"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6vectorI7uint8_tEE9LoggerPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr"></span><span id="tensorrt_llm::runtime::GptSession::GptSession__GptModelConfigCR.WorldConfigCR.ssCR.LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a050978a065f4a2b17ec2d0e9ae97a497"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GptSession</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineFile</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logger</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10GptSessionERK14GptModelConfigRK11WorldConfigRKNSt6stringE9LoggerPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession9getLoggerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession9getLoggerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession9getLoggerEv"></span><span id="tensorrt_llm::runtime::GptSession::getLoggerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3fa9a3cb9db8934ff06b1df13063b2f3"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getLogger</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getLoggerEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv"></span><span id="tensorrt_llm::runtime::GptSession::getBufferManagerC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a992c0a12864a9a30ddccc73819fcad62"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13BufferManagerE" title="tensorrt_llm::runtime::BufferManager"><span class="n"><span class="pre">BufferManager</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getBufferManager</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession16getBufferManagerEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession14getModelConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession14getModelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession14getModelConfigEv"></span><span id="tensorrt_llm::runtime::GptSession::getModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aafeb2d6a1276ed948457924d1a54e744"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getModelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getModelConfigEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv"></span><span id="tensorrt_llm::runtime::GptSession::getWorldConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a8921d43edbec6aa0b9ba6a6894505542"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getWorldConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession14getWorldConfigEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession9getDeviceEv"></span><span id="tensorrt_llm::runtime::GptSession::getDeviceC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a94905fac580c8d0cd5b26b7fe319e853"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession9getDeviceEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime10GptSession15isCudaGraphModeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime10GptSession15isCudaGraphModeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime10GptSession15isCudaGraphModeEv"></span><span id="tensorrt_llm::runtime::GptSession::isCudaGraphModeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a0aa5b90cb9e19019034d913e326d37b8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isCudaGraphMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime10GptSession15isCudaGraphModeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16setCudaGraphModeEb">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16setCudaGraphModeEb"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16setCudaGraphModeEb"></span><span id="tensorrt_llm::runtime::GptSession::setCudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ac9bbe4bb488194510a54740483878c19"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCudaGraphMode</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">value</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16setCudaGraphModeEb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession5setupE8SizeType8SizeType8SizeTypebNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession5setupE8SizeType8SizeType8SizeTypebNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession5setupE8SizeType8SizeType8SizeTypebNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::GptSession::setup__SizeType.SizeType.SizeType.b.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3170f16094aafed96e93723675715a16"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decoderPerRequest</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokensInPagedKvCache</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession5setupE8SizeType8SizeType8SizeTypebNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Initialize buffers for the given sizes. <code class="docutils literal notranslate"><span class="pre">generate</span></code> may be called with batch size and beam width smaller than the setup parameters. </p>
<p><code class="docutils literal notranslate"><span class="pre">maxBatchSize</span></code> will be devided by the number of micro batches to initialize each batch buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptSession::generate__GenerationOutputR.GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a55d90d8c6fccd7de2975d20ded7e4e42"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8generateER16GenerationOutputRK15GenerationInputRK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14KvCacheManagerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14KvCacheManagerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a17734c9adff9c0c3dd37d104c9c20879"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheManager</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="tensorrt_llm::batch_manager"><span class="n"><span class="pre">batch_manager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="tensorrt_llm::batch_manager::kv_cache_manager"><span class="n"><span class="pre">kv_cache_manager</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">KVCacheManager</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession19generateSingleBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession19generateSingleBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession19generateSingleBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptSession::generateSingleBatch__GenerationOutputR.GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a97e3d000bf8d0d49f642228ef659c510"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generateSingleBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19generateSingleBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession18generateMultiBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession18generateMultiBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession18generateMultiBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::GptSession::generateMultiBatch__GenerationOutputR.GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a5ee2425cab580f6fb0d6bd1fb9beedfb"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generateMultiBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16GenerationOutputE" title="tensorrt_llm::runtime::GenerationOutput"><span class="n"><span class="pre">GenerationOutput</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession18generateMultiBatchER16GenerationOutputRK15GenerationInputRK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createContexts__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a5082e4db217c9ac0c80b0aaf341fdac6"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createContexts</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createContextsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createBuffers__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a9a0d4e7b740eff3848b6cb2d15d99608"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createBuffers</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13createBuffersE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createDecoders__SizeType.SizeType.SizeType.nvinfer1::DataType.b.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1afeeed7a6643e35e9a917a045fe6e64f3"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createDecoders</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsType</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decoderPerRequest</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14createDecodersE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeEb8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession21createKvCacheManagersE8SizeType8SizeType8SizeType8SizeTypeNSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession21createKvCacheManagersE8SizeType8SizeType8SizeType8SizeTypeNSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession21createKvCacheManagersE8SizeType8SizeType8SizeType8SizeTypeNSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::GptSession::createKvCacheManagers__SizeType.SizeType.SizeType.SizeType.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a824b4259610fd481825be38792db3326"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createKvCacheManagers</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numMicroBatches</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokensInPagedKvCache</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession21createKvCacheManagersE8SizeType8SizeType8SizeType8SizeTypeNSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::createCustomAllReduceWorkspace__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a761e1bcaecfcd45d43b9737a97f88ef4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">createCustomAllReduceWorkspace</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession30createCustomAllReduceWorkspaceE8SizeType8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16decoderStepAsyncERN7ITensor9SharedPtrERN7ITensor9SharedPtrE8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16decoderStepAsyncERN7ITensor9SharedPtrERN7ITensor9SharedPtrE8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16decoderStepAsyncERN7ITensor9SharedPtrERN7ITensor9SharedPtrE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::decoderStepAsync__ITensor::SharedPtrR.ITensor::SharedPtrR.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1afe77fa5295c0902fa6a6a21a0fac1a02"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoderStepAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">newTokens</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">decoderStep</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16decoderStepAsyncERN7ITensor9SharedPtrERN7ITensor9SharedPtrE8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Execute decoder on last PP rank, receive decoder output on other PP ranks. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::shouldStopSync__SizeType.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a4d2ef8994105f56856cd3e44666611e2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shouldStopSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14shouldStopSyncE8SizeType8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Synchronize with the decoder and return the <code class="docutils literal notranslate"><span class="pre">shouldStop</span></code> flag. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17finalizeOutputIdsER7ITensor8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17finalizeOutputIdsER7ITensor8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17finalizeOutputIdsER7ITensor8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::finalizeOutputIds__ITensorR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ae8296224f803d653980ffdbddae574e7"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finalizeOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17finalizeOutputIdsER7ITensor8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Collect final output ids on last PP rank and send them to first PP rank. </p>
<p>Receives are asynchronous on host, so synchronization is required before access. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::kvCacheAddSequences__SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aaf8b7e1bce5d7aafeb3864a5c4bb56ef"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheAddSequences</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19kvCacheAddSequencesE8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13initNewTokensERK15GenerationInputRK14SamplingConfig8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13initNewTokensERK15GenerationInputRK14SamplingConfig8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13initNewTokensERK15GenerationInputRK14SamplingConfig8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::initNewTokens__GenerationInputCR.SamplingConfigCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3da5c6695d6822c3e5e585754523c201"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">initNewTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">microBatchId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13initNewTokensERK15GenerationInputRK14SamplingConfig8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession12mModelConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession12mModelConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession12mModelConfigE"></span><span id="tensorrt_llm::runtime::GptSession::mModelConfig__GptModelConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a7c9df79575f744947edb70aabeceb1ae"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14GptModelConfigE" title="tensorrt_llm::runtime::GptModelConfig"><span class="n"><span class="pre">GptModelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mModelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mModelConfigE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession12mWorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession12mWorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession12mWorldConfigE"></span><span id="tensorrt_llm::runtime::GptSession::mWorldConfig__WorldConfigC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a718d9218f4e18056a985e79d6d301747"></span><span class="k"><span class="pre">const</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession12mWorldConfigE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession7mDeviceE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession7mDeviceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession7mDeviceE"></span><span id="tensorrt_llm::runtime::GptSession::mDevice__i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a557d94b8bbfbdc168c417eeae307c25a"></span><span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDevice</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="o"><span class="pre">-</span></span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mDeviceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession13mPipelineCommE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession13mPipelineCommE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession13mPipelineCommE"></span><span id="tensorrt_llm::runtime::GptSession::mPipelineComm__std::shared_ptr:NcclCommunicator:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1afd92278f3aa9e57d02c58f5c5cb1de3a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">NcclCommunicator</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineComm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession13mPipelineCommE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession11mCommStreamE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession11mCommStreamE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession11mCommStreamE"></span><span id="tensorrt_llm::runtime::GptSession::mCommStream__std::shared_ptr:CudaStream:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a14480c0db4c1615ed788836285a3c1e7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommStream</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession11mCommStreamE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession10mCommEventE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession10mCommEventE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession10mCommEventE"></span><span id="tensorrt_llm::runtime::GptSession::mCommEvent__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a0916e45f7ddd9ac0a27eec4897260802"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommEvent</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession10mCommEventE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE"></span><span id="tensorrt_llm::runtime::GptSession::mDecoderMaxSequenceLength__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3eb70f53bb0cbc5c6e032c37520b101c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoderMaxSequenceLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession25mDecoderMaxSequenceLengthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession7mLoggerE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession7mLoggerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession7mLoggerE"></span><span id="tensorrt_llm::runtime::GptSession::mLogger__LoggerPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a4bd2d4d4f201bf17a87ed40c4e29ce55"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9LoggerPtrE" title="tensorrt_llm::runtime::GptSession::LoggerPtr"><span class="n"><span class="pre">LoggerPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogger</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession7mLoggerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8mRuntimeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8mRuntimeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8mRuntimeE"></span><span id="tensorrt_llm::runtime::GptSession::mRuntime__std::shared_ptr:TllmRuntime:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1aa97fc0a02f3f28e3e2121e2e9bc10c56"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">TllmRuntime</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRuntime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mRuntimeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16mNumMicroBatchesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16mNumMicroBatchesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16mNumMicroBatchesE"></span><span id="tensorrt_llm::runtime::GptSession::mNumMicroBatches__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a25c4baa3f042dae7e97ea316a746a538"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumMicroBatches</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16mNumMicroBatchesE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession9mDecodersE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession9mDecodersE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession9mDecodersE"></span><span id="tensorrt_llm::runtime::GptSession::mDecoders__std::vector:std::shared_ptr:IStatefulGptDecoder::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1ab3287006aede8e62f11db44d5f8997e6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="tensorrt_llm::runtime::IStatefulGptDecoder"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecoders</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession9mDecodersE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession8mBuffersE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession8mBuffersE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession8mBuffersE"></span><span id="tensorrt_llm::runtime::GptSession::mBuffers__std::vector:std::shared_ptr:RuntimeBuffers::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a96b0baf2000c1616dcdcecf836cdd143"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">RuntimeBuffers</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBuffers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession8mBuffersE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession16mKvCacheManagersE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession16mKvCacheManagersE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession16mKvCacheManagersE"></span><span id="tensorrt_llm::runtime::GptSession::mKvCacheManagers__std::vector:std::shared_ptr:KvCacheManager::"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a09318539e6548be6001a61d0fecb69fa"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14KvCacheManagerE" title="tensorrt_llm::runtime::GptSession::KvCacheManager"><span class="n"><span class="pre">KvCacheManager</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mKvCacheManagers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession16mKvCacheManagersE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession15mReceivedEventsE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession15mReceivedEventsE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession15mReceivedEventsE"></span><span id="tensorrt_llm::runtime::GptSession::mReceivedEvents__std::vector:CudaEvent:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a9253a1bccec0b1ab57bab7b7d5b6f5bb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mReceivedEvents</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession15mReceivedEventsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE"></span><span id="tensorrt_llm::runtime::GptSession::mCudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a1bf104808d61798aeba1c87beeb2c101"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphMode</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession14mCudaGraphModeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE"></span><span id="tensorrt_llm::runtime::GptSession::mCudaGraphInstances__std::array:CudaGraphExecutor.2:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1a3c80743c41fb287fbc0eb5723bbefa7b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE" title="tensorrt_llm::runtime::GptSession::CudaGraphExecutor"><span class="n"><span class="pre">CudaGraphExecutor</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphInstances</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession19mCudaGraphInstancesE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaGraphExecutor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a0f76e8397ca3097bc3451e2251c4d6bd"></span><span class="sig-name descname"><span class="n"><span class="pre">CudaGraphExecutor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor17CudaGraphExecutorEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::~CudaGraphExecutor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a0d609c019d81064a9cdf235352d7c25f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~CudaGraphExecutor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutorD0Ev" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::hasInstance"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a11ce3f651b7e0c1dbba82460a85cfd3b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">hasInstance</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor11hasInstanceEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::clear"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a6bd0bf9ff37bb89cb06d03fb19b2de59"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">clear</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor5clearEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::prepareNextGraph__TllmRuntimeCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1ae8bcb9ec8982af90576b5fc24b6f5599"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">prepareNextGraph</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">TllmRuntime</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">runtime</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">nextContextId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16prepareNextGraphERK11TllmRuntime8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::launch__CudaStreamCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a9d32b4b5046b27364c3e265b433b62a4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">launch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6launchERK10CudaStream" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16cudaGraphExecPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16cudaGraphExecPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16cudaGraphExecPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a6e7ee051d29b4a35c5ecb280404cef77"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cudaGraphExecPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">cudaGraphExec_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16cudaGraphExecPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::create__cudaGraph_tCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a368e5d334bee7c4afd9085fdbe98f02e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">create</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaGraph_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">graph</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6createERK11cudaGraph_t" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::update__cudaGraph_tCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1a776709d610ef7122a55d1a9dd211acb9"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">update</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">cudaGraph_t</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">graph</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor6updateERK11cudaGraph_t" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::uploadToStream__CudaStreamCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1af2cd5b05bdcf087f91696bc83daa90e0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">uploadToStream</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor14uploadToStreamERK10CudaStream" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE">
<span id="_CPPv3N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE"></span><span id="_CPPv2N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE"></span><span id="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::mInstance__cudaGraphExecPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1GptSession_1_1CudaGraphExecutor_1afb0fac5805c84a909296f85363c505ed"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor16cudaGraphExecPtrE" title="tensorrt_llm::runtime::GptSession::CudaGraphExecutor::cudaGraphExecPtr"><span class="n"><span class="pre">cudaGraphExecPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInstance</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10GptSession17CudaGraphExecutor9mInstanceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime5utilsE">
<span id="_CPPv3N12tensorrt_llm7runtime5utilsE"></span><span id="_CPPv2N12tensorrt_llm7runtime5utilsE"></span><span id="tensorrt_llm::runtime::utils"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1utils"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">utils</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime5utilsE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE"></span><span id="tensorrt_llm::runtime::utils::loadEngine__ssCR"></span><span class="target" id="gptSession_8h_1ae397b7fd72675fbc97c0e662cc71b5eb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">loadEngine</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">enginePath</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime5utils10loadEngineERKNSt6stringE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="ibuffer-h">
<h2>iBuffer.h<a class="headerlink" href="#ibuffer-h" title="Permalink to this heading"></a></h2>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE" title="tensorrt_llm::runtime::MemoryType::kGPU"><span class="n"><span class="pre">kGPU</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kGPU:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4_1adc272a863661fdf8ea08ad956ebe5975"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;GPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kGPUEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE" title="tensorrt_llm::runtime::MemoryType::kCPU"><span class="n"><span class="pre">kCPU</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kCPU:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4_1ad18d7d70b64c48821a5928c3eb1b6d0e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;CPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType4kCPUEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE" title="tensorrt_llm::runtime::MemoryType::kPINNED"><span class="n"><span class="pre">kPINNED</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE"></span><span id="tensorrt_llm::runtime::MemoryTypeString:MemoryType::kPINNED:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4_1a661ffce25b1fb3d0f97348ef5e92e6b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;PINNED&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16MemoryTypeStringIN10MemoryType7kPINNEDEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kFLOAT</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aae52e3ccf7e29b666cd03fec6539e026"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kFLOATEE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kHALF_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kHALF</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1adb4c7442a60ad648dc57760e15affff4"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">half</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kHALFEE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT8_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1aaa4acb1449b214520aa455642c7d95c8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kINT8EE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a4f40534926f2d5f89e2dfac85ff27c2b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a7d116567f78612c70c940df5fe74f981"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1afc8c6e195c4db3b75d5bbe02edc8142e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT32EXL1EEE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a5963e497e488b6617c842048bcb4a07c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kINT64EXL1EEE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedEE">
<span id="_CPPv3I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedEE"></span><span id="_CPPv2I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kBOOL</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedEE" title="tensorrt_llm::runtime::CppDataType&lt;nvinfer1::DataType::kBOOL, kUnsigned&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1aabba6c53746defcf62a8448c7511cdcb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType5kBOOLE9kUnsignedE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedEE">
<span id="_CPPv3I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedEE"></span><span id="_CPPv2I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedEE" title="tensorrt_llm::runtime::CppDataType&lt;nvinfer1::DataType::kUINT8, kUnsigned&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_bEN12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1ad9bc02ed66999f1a5f09850f36dfc8ad"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeIN8nvinfer18DataType6kUINT8E9kUnsignedE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4_1a996ff7940394a814ae164728312bf5ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt6int8_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4_1a10403a956a1d979e0cd10c8034e9f3e1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int32_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4_1a96b97cf0edb4ad984dd69278acfabd10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint32_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::int64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4_1af2839adf78453826b3a7f8422c9130fb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7int64_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4_1a6d279398fbfedbbfffd0f7fbd1eb80b8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt8uint64_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:std::uint8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4_1a60d418bc6cea9f845ee353d78f0c3d66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeINSt7uint8_tEE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime18PointerElementTypeE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime18PointerElementTypeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a6b78d5482ec51a801a9fe54db6eaa0f9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PointerElementType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_reference_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">element_type</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-enums">Enums</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryTypeE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryType</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType4kGPUE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType4kGPUE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869a37ae524b76587efa776affdc5cdf2ac1"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kGPUE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType4kCPUE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType4kCPUE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869a2ce310327f474afc9f6774faa2f57903"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType4kCPUE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE">
<span id="_CPPv3N12tensorrt_llm7runtime10MemoryType7kPINNEDE"></span><span id="_CPPv2N12tensorrt_llm7runtime10MemoryType7kPINNEDE"></span><span class="target" id="iBuffer_8h_1a57051a59520fc335a72e6fd1d08fa869ae3278bcaa387e6baeef9b80c1e61c35a"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kPINNED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10MemoryType7kPINNEDE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime16constPointerCastERKNSt10shared_ptrI1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime16constPointerCastERKNSt10shared_ptrI1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1ad7234909a7d00791f3d22a797078e9de"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_const_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">constPointerCast</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERKNSt10shared_ptrI1TEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE">
<span id="_CPPv3I00EN12tensorrt_llm7runtime16constPointerCastERRNSt10unique_ptrI1T1DEE"></span><span id="_CPPv2I00EN12tensorrt_llm7runtime16constPointerCastERRNSt10unique_ptrI1T1DEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">D</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a1398d57cec6aacd5be041575ccb140fb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_const_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">constPointerCast</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="tensorrt_llm::runtime::constPointerCast::D"><span class="n"><span class="pre">D</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4I00EN12tensorrt_llm7runtime16constPointerCastENSt10shared_ptrINSt14remove_const_tI1TEEEERRNSt10unique_ptrI1T1DEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10bufferCastERK7IBuffer"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10bufferCastERK7IBuffer"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1aa39a24d2db1c10a205c75aa2c6a8f069"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer" title="tensorrt_llm::runtime::bufferCast::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">bufferCast</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEPK1TRK7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer">
<span id="_CPPv3I0EN12tensorrt_llm7runtime10bufferCastER7IBuffer"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime10bufferCastER7IBuffer"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="iBuffer_8h_1a2f29317ef105d176296ef6b5518bcddf"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer" title="tensorrt_llm::runtime::bufferCast::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">bufferCast</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime10bufferCastEP1TR7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.IBufferCR"></span><span class="target" id="iBuffer_8h_1a4363785cd968e4e0cd1933cba37850b9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Utility function to print a buffer. </p>
</dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>A wrapper around <code class="docutils literal notranslate"><span class="pre">nvinfer1::DataType</span></code> that provides a support for pointer types. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb"></span><span id="tensorrt_llm::runtime::BufferDataType::BufferDataType__nvinfer1::DataType.b.bCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a3cdb7e5d96b677f670d211cd3f0d2029"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferDataType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">_unsigned</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pointer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType14BufferDataTypeEN8nvinfer18DataTypeEbb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv"></span><span id="tensorrt_llm::runtime::BufferDataType::castto-nvinfer1::DataType-operatorCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1ae0e234f2e49d57ea1ec39bf76ef5f843"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataTypecvN8nvinfer18DataTypeEEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv"></span><span id="tensorrt_llm::runtime::BufferDataType::getDataTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a375ef9858082b42a89a8fbddeb758836"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType11getDataTypeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType9isPointerEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType9isPointerEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType9isPointerEv"></span><span id="tensorrt_llm::runtime::BufferDataType::isPointerCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a4f456474dbc06fcfb4c5fdd4f30c681b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isPointer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType9isPointerEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv"></span><span id="tensorrt_llm::runtime::BufferDataType::isUnsignedCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a84da512a92aa9b9ba7498d8eface36d1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUnsigned</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType10isUnsignedEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14BufferDataType7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14BufferDataType7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14BufferDataType7getSizeEv"></span><span id="tensorrt_llm::runtime::BufferDataType::getSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a0e1c8d7eb345611735b5c8998efc1b29"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14BufferDataType7getSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType::kTrtPointerType__auto"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a10497366a0660dd533bd6bac96a65e97"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTrtPointerType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType9mDataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType9mDataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType9mDataTypeE"></span><span id="tensorrt_llm::runtime::BufferDataType::mDataType__nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a57916add2426171d7066d14df4e796d8"></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mDataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType9mUnsignedE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType9mUnsignedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType9mUnsignedE"></span><span id="tensorrt_llm::runtime::BufferDataType::mUnsigned__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a675fe3124bb2de4dd047ce69a1a5a88b"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mUnsigned</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType9mUnsignedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14BufferDataType8mPointerE">
<span id="_CPPv3N12tensorrt_llm7runtime14BufferDataType8mPointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime14BufferDataType8mPointerE"></span><span id="tensorrt_llm::runtime::BufferDataType::mPointer__b"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferDataType_1a494f3b4c8e08ef43209cac0e2f114e0c"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPointer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType8mPointerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime11BufferRangeE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime11BufferRangeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferRange</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange10value_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange10value_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a577abddadeb9927120873f9d07325e37"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange9size_typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange9size_typeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ac52032549a72c5078e9dea0d731626ab"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size_type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange9referenceE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange9referenceE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1acb95456868b359618379a83adc048eaf"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reference</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="tensorrt_llm::runtime::BufferRange::value_type"><span class="n"><span class="pre">value_type</span></span></a><span class="p"><span class="pre">&amp;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange15const_referenceE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange15const_referenceE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aed057be953bd7fb90a48f5a7c3ae1692"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_reference</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange10value_typeE" title="tensorrt_llm::runtime::BufferRange::value_type"><span class="n"><span class="pre">value_type</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange7pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange7pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a286cbcbcb22a4f81104c9124130118c3"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange13const_pointerE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange13const_pointerE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aad04e26d81c56579e3d6a69ac5fa7412"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_pointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange8iteratorE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange8iteratorE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aeb69cf6a643d84e5c7d051bee4311869"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iterator</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange7pointerE" title="tensorrt_llm::runtime::BufferRange::pointer"><span class="n"><span class="pre">pointer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange14const_iteratorE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange14const_iteratorE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a36e1f97c5d72f04e004f1624a5709986"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">const_iterator</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange13const_pointerE" title="tensorrt_llm::runtime::BufferRange::const_pointer"><span class="n"><span class="pre">const_pointer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer"></span><span id="tensorrt_llm::runtime::BufferRange::BufferRange__IBufferR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a28e8d9ed5250f967aa8f06317074af7b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferRange</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange11BufferRangeER7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5beginEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="tensorrt_llm::runtime::BufferRange::begin"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aadf490250d5ce134a93e32ee11a34980"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="tensorrt_llm::runtime::BufferRange::iterator"><span class="n"><span class="pre">iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">begin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5beginEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange3endEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="tensorrt_llm::runtime::BufferRange::end"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aefcb27b14d94348a6fad611d64fbac92"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange8iteratorE" title="tensorrt_llm::runtime::BufferRange::iterator"><span class="n"><span class="pre">iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">end</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange3endEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange5beginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange5beginEv"></span><span id="tensorrt_llm::runtime::BufferRange::beginC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a7fd5ff6c02c395fa4237a0eb1d9330cf"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">begin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange5beginEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange3endEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange3endEv"></span><span id="tensorrt_llm::runtime::BufferRange::endC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a758833712fb9f5aac0ec22591d767fee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">end</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange3endEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange6cbeginEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="tensorrt_llm::runtime::BufferRange::cbegin"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1aad76274f379a648c172fa9f064f88d62"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cbegin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange6cbeginEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange4cendEv">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="tensorrt_llm::runtime::BufferRange::cend"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a28a237e06e8971046f0fd4d7d9011fff"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cend</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange4cendEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange6cbeginEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange6cbeginEv"></span><span id="tensorrt_llm::runtime::BufferRange::cbeginC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a8e5dfa0b4e150b5fdc0ce78af3c5d243"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cbegin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange6cbeginEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange4cendEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange4cendEv"></span><span id="tensorrt_llm::runtime::BufferRange::cendC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1a5df247c962064585ba1c47242a562dea"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange14const_iteratorE" title="tensorrt_llm::runtime::BufferRange::const_iterator"><span class="n"><span class="pre">const_iterator</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cend</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4cendEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRange4sizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRange4sizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRange4sizeEv"></span><span id="tensorrt_llm::runtime::BufferRange::sizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1add27991a49b2ff4b17b187711cd9eba6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">size</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRange4sizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRangeixE9size_type">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="tensorrt_llm::runtime::BufferRange::subscript-operator__size_type"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ab65060237344abbc8db90f2062abb3ab"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9referenceE" title="tensorrt_llm::runtime::BufferRange::reference"><span class="n"><span class="pre">reference</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">[]</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRangeixE9size_type" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11BufferRangeixE9size_type">
<span id="_CPPv3NK12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="_CPPv2NK12tensorrt_llm7runtime11BufferRangeixE9size_type"></span><span id="tensorrt_llm::runtime::BufferRange::subscript-operator__size_typeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1acc8eff6f7541d117de3e09730425a430"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange15const_referenceE" title="tensorrt_llm::runtime::BufferRange::const_reference"><span class="n"><span class="pre">const_reference</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">[]</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11BufferRangeixE9size_type" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5mDataE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5mDataE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5mDataE"></span><span id="tensorrt_llm::runtime::BufferRange::mData__TP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1afd541e9707bd79322c5290261787902e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11BufferRangeE" title="tensorrt_llm::runtime::BufferRange::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">mData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mDataE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11BufferRange5mSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime11BufferRange5mSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11BufferRange5mSizeE"></span><span id="tensorrt_llm::runtime::BufferRange::mSize__size_type"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1BufferRange_1ae18e3afe46fa34b6f3a8d5bfa7911bfc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange9size_typeE" title="tensorrt_llm::runtime::BufferRange::size_type"><span class="n"><span class="pre">size_type</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11BufferRange5mSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime11CppDataTypeE">
<span id="_CPPv3I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime11CppDataTypeE"></span><span id="_CPPv2I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime11CppDataTypeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDataType</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kIsUnsigned</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kIsPointer</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><a class="headerlink" href="#_CPPv4I_N8nvinfer18DataTypeE_b_bEN12tensorrt_llm7runtime11CppDataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>For converting a TensorRT data type to a C++ data type. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE">
<span id="_CPPv3I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE"></span><span id="_CPPv2I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXtrueEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDataType</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnsigned</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01kDataType_00_01kUnsigned_00_01true_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CppDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::CppDataType&lt;kDataType, kUnsigned, true&gt;::kDataType"><span class="n"><span class="pre">kDataType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::CppDataType&lt;kDataType, kUnsigned, true&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEE4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEE4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXtrueEE4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01kDataType_00_01kUnsigned_00_01true_01_4_1a158884a279cc456ecfef4acad4f260a8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">CppDataType</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::CppDataType&lt;kDataType, kUnsigned, true&gt;::kDataType"><span class="n"><span class="pre">kDataType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4I_N8nvinfer18DataTypeE_bEN12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEEE" title="tensorrt_llm::runtime::CppDataType&lt;kDataType, kUnsigned, true&gt;::kUnsigned"><span class="n"><span class="pre">kUnsigned</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">type</span></span><span class="p"><span class="pre">*</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11CppDataTypeI9kDataType9kUnsignedXL1EEE4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;bool</span> <span class="pre">kUnsigned&gt;</span> <span class="pre">kBOOL,</span> <span class="pre">kUnsigned</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError4typeE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError4typeE"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kBOOL_00_01kUnsigned_01_4_1aabba6c53746defcf62a8448c7511cdcb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError4typeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kFLOAT</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kFLOAT_01_4_1aae52e3ccf7e29b666cd03fec6539e026"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kHALF_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kHALF</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kHALF_01_4_1adb4c7442a60ad648dc57760e15affff4"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">half</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT32</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_01_4_1a4f40534926f2d5f89e2dfac85ff27c2b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT32,</span> <span class="pre">true</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT32_00_01true_01_4_1afc8c6e195c4db3b75d5bbe02edc8142e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint32_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT64</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_01_4_1a7d116567f78612c70c940df5fe74f981"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT64,</span> <span class="pre">true</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT64_00_01true_01_4_1a5963e497e488b6617c842048bcb4a07c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT8_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kINT8</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kINT8_01_4_1aaa4acb1449b214520aa455642c7d95c8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;bool</span> <span class="pre">kUnsigned&gt;</span> <span class="pre">kUINT8,</span> <span class="pre">kUnsigned</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1CppDataType_3_01nvinfer1_1_1DataType_1_1kUINT8_00_01kUnsigned_01_4_1ad9bc02ed66999f1a5f09850f36dfc8ad"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">type</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferE"></span><span id="tensorrt_llm::runtime::IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref">tensorrt_llm::runtime::ITensor</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer9UniquePtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer9UniquePtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ab5684b52d1990c0a0f70c25d0b6c1465"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniquePtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer9SharedPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer9SharedPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2f0c3696d87a3887b38da126cbef1759"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a3892842dc7c9dcc1cad622470ac97999"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniqueConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a8eefcf151b57310d8ca78b8e25c46c39"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14SharedConstPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer8DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer8DataTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a4036935d2bbb442bf9d071df2eff24bd"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4dataEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="tensorrt_llm::runtime::IBuffer::data"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a1b29144573f04c11a8c59576a2ddca4f"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer4dataEv"></span><span id="tensorrt_llm::runtime::IBuffer::dataC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a49f4130965beadb2b82a0b6bcd50cd11"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::data__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ada0b7259447dab20f65e144030f55cef"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a pointer to the underlying array at a given element index. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::data__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a797f41a093bb00596edaf60f329f66e1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">index</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer4dataENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a pointer to the underlying array at a given element index. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer7getSizeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getSizeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ab280c88977e5cf7d3c3ab88e29dcbe62"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7getSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the size (in number of elements) of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv"></span><span id="tensorrt_llm::runtime::IBuffer::getSizeInBytesC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a3121c0295ee572bb45067bba2d35defa"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizeInBytes</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer14getSizeInBytesEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the size (in bytes) of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer11getCapacityEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer11getCapacityEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer11getCapacityEv"></span><span id="tensorrt_llm::runtime::IBuffer::getCapacityC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a33c48c142ee4ee1c1537d1bef5a7ed16"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCapacity</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getCapacityEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the capacity of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getDataTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a006dc512c6b37d582a2e825249c4a3a2"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer11getDataTypeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the data type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv"></span><span id="tensorrt_llm::runtime::IBuffer::getMemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ade5fc6e89a07ad03927a616fdcbb0463"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMemoryType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer13getMemoryTypeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the memory type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::resize__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af1281049c93d0a1d61b8682170761ad4"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">resize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">newSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer6resizeENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7releaseEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7releaseEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7releaseEv"></span><span id="tensorrt_llm::runtime::IBuffer::release"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a51dc4a186d9b315dfe77aacd33677ff7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">release</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7releaseEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Releases the buffer. It will be reset to nullptr. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferD0Ev"></span><span id="tensorrt_llm::runtime::IBuffer::~IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1abeb460077884adc9aec0c351c9ef3637"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">~IBuffer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferD0Ev" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer"></span><span id="tensorrt_llm::runtime::IBuffer::IBuffer__IBufferCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ac9fd5dfcd8486d8f8150fdb470c38055"></span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer" title="tensorrt_llm::runtime::IBuffer::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferERK7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBufferaSERK7IBuffer">
<span id="_CPPv3N12tensorrt_llm7runtime7IBufferaSERK7IBuffer"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBufferaSERK7IBuffer"></span><span id="tensorrt_llm::runtime::IBuffer::assign-operator__IBufferCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a10744c55831bf12db698adbb511d660e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBufferaSERK7IBuffer" title="Permalink to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::slice__SharedPtr.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a6b9aed7e66f2bfc1f3f710d12dbceea6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Creates a sliced view on the underlying <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. The view will have the same data type as <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>buffer</strong> – The buffer to view. </p></li>
<li><p><strong>offset</strong> – The offset of the view. </p></li>
<li><p><strong>size</strong> – The size of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a40f5f57b98c9918b617c1de832390b9c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::slice__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a310d420f890b10b16951eae4b0227d21"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer5sliceE9SharedPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a9ffdc0f354482f983b6a16825ad25109"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr"></span><span id="tensorrt_llm::runtime::IBuffer::view__SharedPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a4fb6a8d4a92376cdea0957a26629f53b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtr" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> which can be independently resized. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>tensor</strong> – The tensor to view. </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::view__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a58bc0fcaa0356cf9e20ce1d06b16c70f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4viewE9SharedPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> with a different size. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>size</strong> – The size of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af28ec3097ffc66614052cef9392265bb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer14UniqueConstPtrE" title="tensorrt_llm::runtime::IBuffer::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7IBuffer4viewE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::wrap__voidP.DataType.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a6cf1157eb353c90dbe12711be9af63f2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wraps the given <code class="docutils literal notranslate"><span class="pre">data</span></code> in an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code>. The <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code> will not own the underlying <code class="docutils literal notranslate"><span class="pre">data</span></code> and cannot be resized beyond <code class="docutils literal notranslate"><span class="pre">capacity</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>data</strong> – The data to wrap. </p></li>
<li><p><strong>type</strong> – The data type of the <code class="docutils literal notranslate"><span class="pre">data</span></code>. </p></li>
<li><p><strong>size</strong> – The size of the buffer. </p></li>
<li><p><strong>capacity</strong> – The capacity of the buffer. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>An <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IBuffer"><span class="std std-ref"><span class="pre">IBuffer</span></span></a></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::wrap__voidP.DataType.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1af1fb9e5497bfc31b149ff9477161068d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer8DataTypeE" title="tensorrt_llm::runtime::IBuffer::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer4wrapEPv8DataTypeNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a1f5dd20e613af0bc2a05f0b09343535d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapEP1TNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2e4240d0d7f78a278716c8faccf5e9f9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrP1TNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7IBuffer4wrapERNSt6vectorI1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7IBuffer4wrapERNSt6vectorI1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a84fdf2b484eee3440646edfece5b85b0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9UniquePtrE" title="tensorrt_llm::runtime::IBuffer::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE" title="tensorrt_llm::runtime::IBuffer::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">v</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7IBuffer4wrapE9UniquePtrRNSt6vectorI1TEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv"></span><span id="tensorrt_llm::runtime::IBuffer::memoryType__voidCP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1a2c714d66d0336cac0708008e59dd71cd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">memoryType</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer10memoryTypeEPKv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Determine the memory type of a pointer. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferEv">
<span id="_CPPv3N12tensorrt_llm7runtime7IBuffer7IBufferEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7IBuffer7IBufferEv"></span><span id="tensorrt_llm::runtime::IBuffer::IBuffer"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1ad6ad0acb873c47b64fc8fec029696ac2"></span><span class="sig-name descname"><span class="n"><span class="pre">IBuffer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer7IBufferEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE">
<span id="_CPPv3NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE"></span><span id="_CPPv2NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE"></span><span id="tensorrt_llm::runtime::IBuffer::toBytes__std::sC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IBuffer_1aade9d63190ac92fdb3561a61af5ab2ee"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toBytes</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7IBuffer7toBytesENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns an array index or size in bytes. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryTypeString</span></span></span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime16MemoryTypeStringE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kCPU</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime19PhonyNameDueToError5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime19PhonyNameDueToError5valueE"></span><span id="tensorrt_llm::runtime::PhonyNameDueToError::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kCPU_01_4_1ad18d7d70b64c48821a5928c3eb1b6d0e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;CPU&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19PhonyNameDueToError5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kGPU</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kGPU_01_4_1adc272a863661fdf8ea08ad956ebe5975"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;GPU&quot;</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">kPINNED</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1MemoryTypeString_3_01MemoryType_1_1kPINNED_01_4_1a661ffce25b1fb3d0f97348ef5e92e6b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;PINNED&quot;</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0_bEN12tensorrt_llm7runtime11TRTDataTypeE">
<span id="_CPPv3I0_bEN12tensorrt_llm7runtime11TRTDataTypeE"></span><span id="_CPPv2I0_bEN12tensorrt_llm7runtime11TRTDataTypeE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><a class="headerlink" href="#_CPPv4I0_bEN12tensorrt_llm7runtime11TRTDataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iBuffer.h&gt;</em></div>
<p>For converting a C++ data type to a TensorRT data type. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIbEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIbEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIbEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01bool_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIbEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:b:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01bool_01_4_1a590c40a56a5eabf6069340edc07d84e4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kBOOL</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIbE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIfEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIfEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01float_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIfEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:float:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01float_01_4_1a0a72c60c5a9b946e64a960d1075a5af9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kFLOAT</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIfE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01half_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">half</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeI4halfEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:half:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01half_01_4_1a2febf53869fcfd71830c674652e7c7d4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kHALF</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeI4halfE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int32__t_01_4_1a10403a956a1d979e0cd10c8034e9f3e1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int64__t_01_4_1af2839adf78453826b3a7f8422c9130fb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1int8__t_01_4_1a996ff7940394a814ae164728312bf5ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT8</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint32__t_01_4_1a96b97cf0edb4ad984dd69278acfabd10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT32</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint64__t_01_4_1a6d279398fbfedbbfffd0f7fbd1eb80b8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kINT64</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01std_1_1uint8__t_01_4_1a60d418bc6cea9f845ee353d78f0c3d66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">kUINT8</span></span><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:TP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4_1ae513ee17ebefd7e2c538476736488aa1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::kUnderlyingType"><span class="n"><span class="pre">kUnderlyingType</span></span></a><span class="p"><span class="pre">.</span></span><span class="n"><span class="pre">getDataType</span></span><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::kUnderlyingType"><span class="n"><span class="pre">kUnderlyingType</span></span></a><span class="p"><span class="pre">.</span></span><span class="n"><span class="pre">isUnsigned</span></span><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE"></span><span id="tensorrt_llm::runtime::TRTDataType:TP:::kUnderlyingType__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01T_01_5_01_4_1aafb13ea78bfe5c1ddf1a77093531b0ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUnderlyingType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">{</span></span><span class="n"><span class="pre">TRTDataType</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime11TRTDataTypeIP1TEE" title="tensorrt_llm::runtime::TRTDataType&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIP1TE15kUnderlyingTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE">
<span id="_CPPv3IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE"></span><span id="_CPPv2IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01void_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TRTDataType</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm7runtime11TRTDataTypeIPvEE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE">
<span id="_CPPv3N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE"></span><span id="_CPPv2N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE"></span><span id="tensorrt_llm::runtime::TRTDataType:voidP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1runtime_1_1TRTDataType_3_01void_01_5_01_4_1a63d38011d7c02b6ce399d457537b1a93"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataTypeE" title="tensorrt_llm::runtime::BufferDataType"><span class="n"><span class="pre">BufferDataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14BufferDataType15kTrtPointerTypeE" title="tensorrt_llm::runtime::BufferDataType::kTrtPointerType"><span class="n"><span class="pre">kTrtPointerType</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11TRTDataTypeIPvE5valueE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="igptdecoderbatch-h">
<h2>iGptDecoderBatch.h<a class="headerlink" href="#igptdecoderbatch-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatchE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatchE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoderBatch</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="tensorrt_llm::runtime::IStatefulGptDecoder"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatchE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iGptDecoderBatch.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1GptDecoderBatch"><span class="std std-ref">tensorrt_llm::runtime::GptDecoderBatch</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a8968dd741ba1d28cddca68d850a62f59"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ac23bb332c54e6c2b665d5f35fc64baf8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1aefe802b00d9cc338c76f2bfc7bafad4c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::newRequest__SizeType.decoder_batch::RequestCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a795d229a59bc66688d9609ce62b40775"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="tensorrt_llm::runtime::decoder_batch::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch10newRequestE8SizeTypeRKN13decoder_batch7RequestERK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd><p>Initialize the decoder at <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code> with a new <code class="docutils literal notranslate"><span class="pre">request</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forwardAsync__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a6c990b4aa463031ee4d4e2724997ae0e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch8TokenPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TokenPtr"><span class="n"><span class="pre">TokenPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch12forwardAsyncERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host process and return the token for synchronization. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forwardSync__decoder_batch::TokenCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a7171eb7513202c09360d4139117c424e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardSync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="tensorrt_llm::runtime::decoder_batch::Token"><span class="n"><span class="pre">Token</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">token</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch11forwardSyncERKN13decoder_batch5TokenE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for the call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> associated with a token to complete. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::forward__decoder_batch::OutputR.decoder_batch::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ab7e07276f33bfa6ef3c90baa49ee6eec"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="tensorrt_llm::runtime::decoder_batch::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="tensorrt_llm::runtime::decoder_batch"><span class="n"><span class="pre">decoder_batch</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="tensorrt_llm::runtime::decoder_batch::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch7forwardERN13decoder_batch6OutputERKN13decoder_batch5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests and wait for completion on the host. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a56bbbba5d310262f3614c9ca16d6e1d7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getOutputIdsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch17getFinalOutputIdsE8SizeType">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch17getFinalOutputIdsE8SizeType"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch17getFinalOutputIdsE8SizeType"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getFinalOutputIds__SizeTypeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a5b4e85c580d52fed45a727a4baf0ab0e"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinalOutputIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchIdx</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch17getFinalOutputIdsE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Execute postProcessRequest and returns OutputIds for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>. Result will only be available after event returned </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request <code class="docutils literal notranslate"><span class="pre">batchIdx</span></code>, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch16getFinishedBeamsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch16getFinishedBeamsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch16getFinishedBeamsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getFinishedBeamsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1ae3ea5080726f1ce7b14961eca6827e25"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinishedBeams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch16getFinishedBeamsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth], marks finished requests (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch16getOutputLengthsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch16getOutputLengthsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch16getOutputLengthsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getOutputLengthsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a8f082a6761daadd109c357f30c501433"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputLengths</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch16getOutputLengthsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth], total sequence lengths (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1aaa0b5e6c02b7043c3b29a1d8ca68df76"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch11getFinishedEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize (actual)], marks finished requests (per batch) </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getCumLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1afa130a3df00b7d86e280f5caf017ab9d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCumLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch14getCumLogProbsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth], cumulative log probabilities (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getParentIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a1a37b811b0cc2a98507b57109844a06c"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch9TensorPtrE" title="tensorrt_llm::runtime::IGptDecoderBatch::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParentIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch12getParentIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::getNbStepsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1a935e778ac2c14276d940e09b9411e8ed"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbSteps</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime16IGptDecoderBatch10getNbStepsEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv">
<span id="_CPPv3N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv"></span><span id="_CPPv2N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv"></span><span id="tensorrt_llm::runtime::IGptDecoderBatch::IGptDecoderBatch"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch_1abfc9e04a4f042f3c0220c84cff407ff3"></span><span class="sig-name descname"><span class="n"><span class="pre">IGptDecoderBatch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime16IGptDecoderBatch16IGptDecoderBatchEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batchE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batchE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batchE"></span><span id="tensorrt_llm::runtime::decoder_batch"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1decoder__batch"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoder_batch</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batchE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch6OutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch6OutputE"></span><span class="target" id="iGptDecoderBatch_8h_1a575658a5bf5a41310da470e54bf965d9"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch6OutputE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5InputE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input4BaseE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input4BaseE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input4BaseE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1afd98890b16c1fb49efd955e2f8cc74b5"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Base</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input4BaseE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtr"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a818425caddd990a62547b8cd80155f3b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">TensorPtr</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtrRKNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtrRKNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtrRKNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::Input__TensorPtr.std::vector:b:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a036ae2c032d83f9fa028bf7b1e56f72e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">TensorPtr</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">active</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input5InputE9TensorPtrRKNSt6vectorIbEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6activeE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Input6activeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Input6activeE"></span><span id="tensorrt_llm::runtime::decoder_batch::Input::active__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input_1a85bb8bda589fcebf0c19994d76a26968"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">active</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Input6activeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7RequestE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7RequestE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7RequestE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a016b305f614d064ef624b5798d53dfc7"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request7RequestE9TensorPtrNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request7RequestE9TensorPtrNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request7RequestE9TensorPtrNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::Request__TensorPtr.std::optional:SizeType:.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1adc776675e0d99442e16cba4c39e7f7ab"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ids</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNewTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request7RequestE9TensorPtrNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request3idsE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request3idsE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request3idsE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::ids__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a228f26f47eae229c4664895d797b77d1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ids</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request3idsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::maxNewTokens__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1a5ab1bf54050a431e7c26df7451f2f0f1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNewTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12maxNewTokensE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request5endIdE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request5endIdE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request5endIdE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::endId__std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1ac1477f03a617c556bb264121200fcc0e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">endId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request5endIdE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::embeddingBias__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1acbedf401f2f77534aa355f3ecbb0b755"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">embeddingBias</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13embeddingBiasE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::badWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1af18cc1700b613308c14d492e69ab76e9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">badWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request12badWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE"></span><span id="tensorrt_llm::runtime::decoder_batch::Request::stopWordsList__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Request_1addd0a66af57e7186287ac7e5853c0c9d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request9TensorPtrE" title="tensorrt_llm::runtime::decoder_batch::Request::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stopWordsList</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch7Request13stopWordsListE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5TokenE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5TokenE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Token</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5TokenE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::Token__CudaEventRR.std::vector:b:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1ab35d80ddc444d443a35ae642d3e9f862"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Token</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">event</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">active</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5TokenERR9CudaEventRKNSt6vectorIbEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5eventE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token5eventE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token5eventE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::event__CudaEvent"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1a8a4fa5d069ee58377e79ddde0955dbde"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime9CudaEventE" title="tensorrt_llm::runtime::CudaEvent"><span class="n"><span class="pre">CudaEvent</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">event</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token5eventE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token6activeE">
<span id="_CPPv3N12tensorrt_llm7runtime13decoder_batch5Token6activeE"></span><span id="_CPPv2N12tensorrt_llm7runtime13decoder_batch5Token6activeE"></span><span id="tensorrt_llm::runtime::decoder_batch::Token::active__std::vector:b:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Token_1a92563b2a16378f689f9d600d8712f8fd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">active</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13decoder_batch5Token6activeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="istatefulgptdecoder-h">
<h2>iStatefulGptDecoder.h<a class="headerlink" href="#istatefulgptdecoder-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoderE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoderE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;iStatefulGptDecoder.h&gt;</em></div>
<p>GPT decoder class with support for in-flight batching. </p>
<p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IGptDecoderBatch"><span class="std std-ref">tensorrt_llm::runtime::IGptDecoderBatch</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a310eb1dafd035c945176ea533ec74588"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder13CudaStreamPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a3987d8ce3a11babf865981af9d4ea18e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::setup__SizeType.SizeType.SizeType.nvinfer1::DataType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a40b70b01f26c50f974a46163300c6d9a"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setup</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxSequenceLength</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dtype</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder5setupE8SizeType8SizeType8SizeTypeN8nvinfer18DataTypeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Setup the decoder before calling <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a65d6bb6723258d57f5d58483749c2004"><span class="std std-ref"><span class="pre">forward()</span></span></a></code>, also calls reshapeBuffers. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK14SamplingConfig"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::newBatch__GenerationInputCR.SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a6dfda7e2cc24d12a387dbef8801b2211"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newBatch</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime15GenerationInputE" title="tensorrt_llm::runtime::GenerationInput"><span class="n"><span class="pre">GenerationInput</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inputs</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="tensorrt_llm::runtime::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder8newBatchERK15GenerationInputRK14SamplingConfig" title="Permalink to this definition"></a><br /></dt>
<dd><p>Initialize the decoder with new batch of inputs. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::forwardAsync__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a7bb51780e6e9a6c41de214ffac8b07be"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forwardAsync</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder12forwardAsyncERN7decoder6OutputERKN7decoder5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests without blocking the host thread. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder14isFinishedSyncEv">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder14isFinishedSyncEv"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder14isFinishedSyncEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::isFinishedSync"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a4840aa6124a2590af2b575faf84bfbf7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFinishedSync</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder14isFinishedSyncEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wait for the last call to <code class="docutils literal notranslate"><span class="pre">forwardAsync</span></code> to complete and return whether all sequences have finished. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::forward__decoder::OutputR.decoder::InputCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a65d6bb6723258d57f5d58483749c2004"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">forward</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="tensorrt_llm::runtime::decoder::Output"><span class="n"><span class="pre">Output</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="tensorrt_llm::runtime::decoder"><span class="n"><span class="pre">decoder</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="tensorrt_llm::runtime::decoder::Input"><span class="n"><span class="pre">Input</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">input</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder7forwardERN7decoder6OutputERKN7decoder5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Run one step for all requests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder17getFinalOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder17getFinalOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder17getFinalOutputIdsEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getFinalOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a54e4feeae207c8426d04011e3ca58ef7"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFinalOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder17getFinalOutputIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Gather final results for all requests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getOutputIdsC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a3f98067a78b64a319acb4a8b667e6249"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getOutputIdsEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth, maxSequenceLength], all token ids, on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a2806684ae6ae3a9f4d96042c881d9993"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder12getNewTokensEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[batchSize, beamWidth], latests generated tokens (per beam), on gpu </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::getNbFinishedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a551e560fe7c9120c500f47fa84514d99"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder9TensorPtrE" title="tensorrt_llm::runtime::IStatefulGptDecoder::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNbFinished</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime19IStatefulGptDecoder13getNbFinishedEv" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>[1], number of finished sequences, in pinned host memory </p>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv">
<span id="_CPPv3N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv"></span><span id="_CPPv2N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv"></span><span id="tensorrt_llm::runtime::IStatefulGptDecoder::IStatefulGptDecoder"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IStatefulGptDecoder_1a898fa69e15fa0d26fde3c7f50fdb8881"></span><span class="sig-name descname"><span class="n"><span class="pre">IStatefulGptDecoder</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime19IStatefulGptDecoder19IStatefulGptDecoderEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoderE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoderE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoderE"></span><span id="tensorrt_llm::runtime::decoder"></span><span class="target" id="namespacetensorrt__llm_1_1runtime_1_1decoder"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decoder</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoderE" title="Permalink to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5InputE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5InputE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5InputE"></span><span id="tensorrt_llm::runtime::decoder::Input"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5InputE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Subclassed by <a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1decoder__batch_1_1Input"><span class="std std-ref">tensorrt_llm::runtime::decoder_batch::Input</span></a></p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1a00dc3503cac2813698c97486fba9ca51"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr"></span><span id="tensorrt_llm::runtime::decoder::Input::Input__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1adbf9a04fb7b5c83766b236df5060d24a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Input</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input5InputE9TensorPtr" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input6logitsE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input6logitsE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input6logitsE"></span><span id="tensorrt_llm::runtime::decoder::Input::logits__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1a8b92e3c3e42f408503691dd5fb51e79e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input6logitsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::decoder::Input::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Input_1ae425ecaffbd89893ed70d3ad32c660b1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input9TensorPtrE" title="tensorrt_llm::runtime::decoder::Input::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder5Input16cacheIndirectionE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6OutputE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6OutputE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6OutputE"></span><span id="tensorrt_llm::runtime::decoder::Output"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6OutputE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1a2c4dff939d33fec2fd199699c216ff28"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output6OutputEv">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output6OutputEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output6OutputEv"></span><span id="tensorrt_llm::runtime::decoder::Output::Output"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1af52215c47e70f6af33b6353883c24948"></span><span class="sig-name descname"><span class="n"><span class="pre">Output</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output6OutputEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE"></span><span id="tensorrt_llm::runtime::decoder::Output::cacheIndirection__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1ae9e66b08bf05167a9d16130d0518fc0e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="tensorrt_llm::runtime::decoder::Output::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheIndirection</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output16cacheIndirectionE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE">
<span id="_CPPv3N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE"></span><span id="_CPPv2N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE"></span><span id="tensorrt_llm::runtime::decoder::Output::sequenceLengths__TensorPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1decoder_1_1Output_1a25913f9ceca02070a650d1135ba87a72"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output9TensorPtrE" title="tensorrt_llm::runtime::decoder::Output::TensorPtr"><span class="n"><span class="pre">TensorPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceLengths</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7decoder6Output15sequenceLengthsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="itensor-h">
<h2>iTensor.h<a class="headerlink" href="#itensor-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv48nvinfer1">
<span id="_CPPv38nvinfer1"></span><span id="_CPPv28nvinfer1"></span><span id="nvinfer1"></span><span class="target" id="namespacenvinfer1"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">nvinfer1</span></span></span><a class="headerlink" href="#_CPPv48nvinfer1" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.ITensor::ShapeCR"></span><span class="target" id="iTensor_8h_1ab843bef84b07b782c05c682399ee2b1a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERKN7ITensor5ShapeE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Utility function to print a shape. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor"></span><span id="tensorrt_llm::runtime::lshift-operator__osR.ITensorCR"></span><span class="target" id="iTensor_8h_1a2a38821feb0276616dde2d32020ef451"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">output</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimelsERNSt7ostreamERK7ITensor" title="Permalink to this definition"></a><br /></dt>
<dd><p>Utility function to print a tensor with its shape. </p>
</dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensorE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensorE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensorE"></span><span id="tensorrt_llm::runtime::ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9UniquePtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9UniquePtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a0ebcd2ee37abad2960daf1346816f348"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniquePtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9SharedPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9SharedPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a31e67140fadaeb6eaaf096ff37d640ad"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ad199f95dd25dfbef298cc7d1f3655d66"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UniqueConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor14SharedConstPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor14SharedConstPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a591e6b0aa01a0043967262ffdcc3974d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SharedConstPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14SharedConstPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5ShapeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab2e78d36e5cc72b4443e832a46147fcc"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Dims</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime7ITensor8getShapeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime7ITensor8getShapeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime7ITensor8getShapeEv"></span><span id="tensorrt_llm::runtime::ITensor::getShapeC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a9f2d4a0903e581784f5ed94d8ed1d58d"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getShape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime7ITensor8getShapeEv" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the tensor dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::reshape__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a772377264b089406fb1cf743ac3163f3"></span><span class="k"><span class="pre">virtual</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reshape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7reshapeERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd><p>Sets the tensor dimensions. The new size of the tensor will be <code class="docutils literal notranslate"><span class="pre">volume(dims)</span></code></p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensorD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensorD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensorD0Ev"></span><span id="tensorrt_llm::runtime::ITensor::~ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a857da95b81330009b368308ed0c2d3d9"></span><span class="sig-name descname"><span class="n"><span class="pre">~ITensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">override</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensorD0Ev" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor"></span><span id="tensorrt_llm::runtime::ITensor::ITensor__ITensorCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ade4b1d2a214d378f73255357e32bee1f"></span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor" title="tensorrt_llm::runtime::ITensor::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorERK7ITensor" title="Permalink to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensoraSERK7ITensor">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensoraSERK7ITensor"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensoraSERK7ITensor"></span><span id="tensorrt_llm::runtime::ITensor::assign-operator__ITensorCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a943f0acaa8c746fa85980f4e13512a2c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensoraSERK7ITensor" title="Permalink to this definition"></a><br /></dt>
<dd><p>Not allowed to copy. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::squeeze__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a5ca07f1cc675136634c6a435b5b8f8f3"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">squeeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Removes the given <em>unit</em> dimensions from this tensor. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor6volumeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor6volumeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor6volumeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::volume__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a2afffa26551b43982bf51fa4920d4ade"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">volume</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor6volumeERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the volume of the dimensions. Returns -1 if <code class="docutils literal notranslate"><span class="pre">d.nbDims</span> <span class="pre">&lt;</span> <span class="pre">0</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::volumeNonNegative__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aac34c9309ce068d8216a54343348e21d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">volumeNonNegative</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor17volumeNonNegativeERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns the volume of the dimensions. Throws if <code class="docutils literal notranslate"><span class="pre">d.nbDims</span> <span class="pre">&lt;</span> <span class="pre">0</span></code>. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType"></span><span id="tensorrt_llm::runtime::ITensor::squeeze__ShapeCR.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a7c9ddd4f8b1eaa7b1a6276c49b2c8672"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">squeeze</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dim</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7squeezeERK5Shape8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd><p>Removes the given <em>unit</em> dimension from <code class="docutils literal notranslate"><span class="pre">shape</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape to squeeze. </p></li>
<li><p><strong>dim</strong> – The dimension that should be removed (“squeezed”). </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A new shape without the unit dimension. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::slice__SharedPtr.std::s.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aa22bf50eff27dbfc64eed003851f4778"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Creates a sliced view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. The view will have the same data type as <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>offset</strong> – The offset of the view w.r.t. dimension 0 of the tensor. </p></li>
<li><p><strong>size</strong> – The size of the view w.r.t. dimension 0 of the tensor. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">buffer</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tENSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a201a60f7da459ec35f5d750e8d7053c4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tENSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::slice__SharedPtr.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a3bc0148d89f892df6beff0f63ea33e6a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5sliceE9SharedPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tE"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceERR9TConstPtrNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a01d828a32b3f280c1ee64ce5211da20b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">slice</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="tensorrt_llm::runtime::ITensor::slice::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">offset</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor5sliceE14UniqueConstPtrRR9TConstPtrNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::view__IBuffer::SharedPtr.ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1aeb7a31c92f9ebc17e4648ba4fff7e16e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBufferE" title="tensorrt_llm::runtime::IBuffer"><span class="n"><span class="pre">IBuffer</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7IBuffer9SharedPtrE" title="tensorrt_llm::runtime::IBuffer::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">buffer</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewEN7IBuffer9SharedPtrERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">buffer</span></code> (or tensor) with the given shape. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>tensor</strong> – The tensor to view. </p></li>
<li><p><strong>shape</strong> – The shape of the view. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape">
<span id="_CPPv3I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewERR9TConstPtrRK5Shape"></span><span id="_CPPv2I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewERR9TConstPtrRK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TConstPtr</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">enable_if_t</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">is_const_v</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime18PointerElementTypeE" title="tensorrt_llm::runtime::PointerElementType"><span class="n"><span class="pre">PointerElementType</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="tensorrt_llm::runtime::ITensor::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a481831d7c0d45fe462e91b07beef2617"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor14UniqueConstPtrE" title="tensorrt_llm::runtime::ITensor::UniqueConstPtr"><span class="n"><span class="pre">UniqueConstPtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="tensorrt_llm::runtime::ITensor::view::TConstPtr"><span class="n"><span class="pre">TConstPtr</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0_NSt11enable_if_tINSt10is_const_vI18PointerElementTypeI9TConstPtrEEEiEEEN12tensorrt_llm7runtime7ITensor4viewE14UniqueConstPtrRR9TConstPtrRK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr"></span><span id="tensorrt_llm::runtime::ITensor::view__SharedPtr"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a0d748a6fdb8219af29ba918878bf44b2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">view</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4viewE9SharedPtr" title="Permalink to this definition"></a><br /></dt>
<dd><p>Returns a view on the underlying <code class="docutils literal notranslate"><span class="pre">tensor</span></code> which can be independently reshaped. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>tensor</strong> – The tensor to view. </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A view on the <code class="docutils literal notranslate"><span class="pre">tensor</span></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE"></span><span id="tensorrt_llm::runtime::ITensor::wrap__voidP.nvinfer1::DataType.ShapeCR.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1abb208b99fc6a404a13215fe2d43637a4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5ShapeNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd><p>Wraps the given <code class="docutils literal notranslate"><span class="pre">data</span></code> in an <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code>. The <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code> will not own the underlying <code class="docutils literal notranslate"><span class="pre">data</span></code> and cannot be reshaped beyond <code class="docutils literal notranslate"><span class="pre">capacity</span></code>. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>data</strong> – The data to wrap. </p></li>
<li><p><strong>type</strong> – The data type of the <code class="docutils literal notranslate"><span class="pre">data</span></code>. </p></li>
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>capacity</strong> – The capacity of the buffer. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>An <code class="docutils literal notranslate"><a class="reference internal" href="#classtensorrt__llm_1_1runtime_1_1ITensor"><span class="std std-ref"><span class="pre">ITensor</span></span></a></code>. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::wrap__voidP.nvinfer1::DataType.ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a5fdde0b2cc5bb0ee07c042964bb48c85"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">DataType</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor4wrapEPvN8nvinfer18DataTypeERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5ShapeNSt6size_tE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5ShapeNSt6size_tE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a6cb1dc1fe13779d7d0bed2062df2c0e0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">capacity</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5ShapeNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapEP1TRK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a496e23f8dcc29f792a7e0f70fca11973"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrP1TRK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape">
<span id="_CPPv3I0EN12tensorrt_llm7runtime7ITensor4wrapERNSt6vectorI1TEERK5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime7ITensor4wrapERNSt6vectorI1TEERK5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1ab27a7a0ae81205699d2b2bc618367857"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9UniquePtrE" title="tensorrt_llm::runtime::ITensor::UniquePtr"><span class="n"><span class="pre">UniquePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">wrap</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape" title="tensorrt_llm::runtime::ITensor::wrap::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">v</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime7ITensor4wrapE9UniquePtrRNSt6vectorI1TEERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::ITensor::makeShape__std::initializer_list:SizeType:CR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a7ba94f17049ee1d6048ae69be48ecb67"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">makeShape</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">initializer_list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9makeShapeERKNSt16initializer_listI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd><p>A convenience function to create a tensor shape with the given dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor8toStringERK5Shape">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor8toStringERK5Shape"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor8toStringERK5Shape"></span><span id="tensorrt_llm::runtime::ITensor::toString__ShapeCR"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1af9ba06b8cbf144ea573e6767af85bd43"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor5ShapeE" title="tensorrt_llm::runtime::ITensor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor8toStringERK5Shape" title="Permalink to this definition"></a><br /></dt>
<dd><p>A convenience function for converting a tensor shape to a <code class="docutils literal notranslate"><span class="pre">string</span></code>. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-protected-functions">Protected Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorEv">
<span id="_CPPv3N12tensorrt_llm7runtime7ITensor7ITensorEv"></span><span id="_CPPv2N12tensorrt_llm7runtime7ITensor7ITensorEv"></span><span id="tensorrt_llm::runtime::ITensor::ITensor"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1ITensor_1a60742a4752e9d9d5a82fff9e24518953"></span><span class="sig-name descname"><span class="n"><span class="pre">ITensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime7ITensor7ITensorEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="ipcutils-h">
<h2>ipcUtils.h<a class="headerlink" href="#ipcutils-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb">
<span id="_CPPv3N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb"></span><span id="_CPPv2N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb"></span><span id="tensorrt_llm::runtime::setPeerAccess__WorldConfig.b"></span><span class="target" id="ipcUtils_8h_1a16050a9ed921a12d13a273478086f95c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPeerAccess</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enable</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime13setPeerAccessE11WorldConfigb" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemoryE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemoryE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemoryE"></span><span id="tensorrt_llm::runtime::IpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IpcMemory</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9TensorPtrE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a1c17fa36835ec34cb3cef927e85967eb"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime7ITensor9SharedPtrE" title="tensorrt_llm::runtime::ITensor::SharedPtr"><span class="n"><span class="pre">SharedPtr</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9TensorPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE"></span><span id="tensorrt_llm::runtime::IpcMemory::IpcMemory__WorldConfig.std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1ab60128a164106fe08d754357214b5b82"></span><span class="sig-name descname"><span class="n"><span class="pre">IpcMemory</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">worldConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">bufferSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9IpcMemoryE11WorldConfigNSt6size_tE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemoryD0Ev">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemoryD0Ev"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemoryD0Ev"></span><span id="tensorrt_llm::runtime::IpcMemory::~IpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a0f068272bfaeadb2e976b44adf47c484"></span><span class="sig-name descname"><span class="n"><span class="pre">~IpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemoryD0Ev" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv">
<span id="_CPPv3NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv"></span><span id="tensorrt_llm::runtime::IpcMemory::getCommPtrsTensorC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a8bcbdce69d5e3de95a0f742e430f822c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getCommPtrsTensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime9IpcMemory17getCommPtrsTensorEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE"></span><span id="tensorrt_llm::runtime::IpcMemory::FLAGS_SIZE__s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a4146075f3f6beba4eab4a765506be9bd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FLAGS_SIZE</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">kernels</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MAX_ALL_REDUCE_BLOCKS</span></span><span class="w"> </span><span class="o"><span class="pre">*</span></span><span class="w"> </span><span class="k"><span class="pre">sizeof</span></span><span class="p"><span class="pre">(</span></span><span class="n"><span class="pre">uint32_t</span></span><span class="p"><span class="pre">)</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10FLAGS_SIZEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv"></span><span id="tensorrt_llm::runtime::IpcMemory::allocateIpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a68e4ab83a9194bdd5cf59a1d4ee68a80"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocateIpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory17allocateIpcMemoryEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv"></span><span id="tensorrt_llm::runtime::IpcMemory::destroyIpcMemory"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a7b7c035819f96cb37702472ea179c33b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">destroyIpcMemory</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory16destroyIpcMemoryEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE"></span><span id="tensorrt_llm::runtime::IpcMemory::mWorldConfig__WorldConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1aab66f175e30581b311c654816aa85ea4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory12mWorldConfigE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE"></span><span id="tensorrt_llm::runtime::IpcMemory::mCommPtrs__std::vector:voidP:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1aa15ac983a61b79976435b4d067daccc2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommPtrs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory9mCommPtrsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE"></span><span id="tensorrt_llm::runtime::IpcMemory::mBufferSize__std::s"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1a95b1167806fe77f1f22810ac7477d27d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBufferSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory11mBufferSizeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE">
<span id="_CPPv3N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE"></span><span id="_CPPv2N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE"></span><span id="tensorrt_llm::runtime::IpcMemory::mBufferPtr__voidP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1IpcMemory_1adf62ec2d27e1382c5414188fd3982f8b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">mBufferPtr</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime9IpcMemory10mBufferPtrE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="memorycounters-h">
<h2>memoryCounters.h<a class="headerlink" href="#memorycounters-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCountersE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCountersE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCountersE"></span><span id="tensorrt_llm::runtime::MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryCounters</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a5aa02d808601c67ba934f8a0a9534d6e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ab5afaef89516ec27d7934229de37e387"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DiffType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ptrdiff_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a97911bd1f11691ed4a652bf78e049e08"></span><span class="sig-name descname"><span class="n"><span class="pre">MemoryCounters</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters14MemoryCountersEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getGpuC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a09119207f2dfca61e4e7bd5a66c36654"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpu</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getGpuEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getCpuC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1af643b8967c9f99d8efd647d7b3efe281"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCpu</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters6getCpuEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getPinnedC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ae37c0265530ae4fac20d31da5712d3c0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPinned</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters9getPinnedEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getGpuDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a399a7926f7819096b4f29fa04b3290de"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpuDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getGpuDiffEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getCpuDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a62f4e6d3f0dd20c4ec6b35b4b6df0d47"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCpuDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters10getCpuDiffEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv">
<span id="_CPPv3NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getPinnedDiffC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a4571ac85d01e0844b946d57615523ad0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPinnedDiff</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime14MemoryCounters13getPinnedDiffEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateEv8SizeType">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateE8SizeType"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateE8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a891aab52104e74b95138f4bf20c111c2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters8allocateEv8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType"></span><span id="tensorrt_llm::runtime::MemoryCounters::allocate__MemoryType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a0b437c35c4af66d2f2b2379a40c136d6"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8allocateE10MemoryType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateEv8SizeType">
<span id="_CPPv3I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateE8SizeType"></span><span id="_CPPv2I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateE8SizeType"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a6301480be8d33d2a3e7dbd54119467c9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deallocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I_10MemoryTypeEN12tensorrt_llm7runtime14MemoryCounters10deallocateEv8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType"></span><span id="tensorrt_llm::runtime::MemoryCounters::deallocate__MemoryType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a4cbe26fb0d89352c1de2cb6f63220aa2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deallocate</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime10MemoryTypeE" title="tensorrt_llm::runtime::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">memoryType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters10deallocateE10MemoryType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv"></span><span id="tensorrt_llm::runtime::MemoryCounters::getInstance"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a5fd849b9f6dd583910a3aa4acb8ea391"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="tensorrt_llm::runtime::MemoryCounters"><span class="n"><span class="pre">MemoryCounters</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getInstance</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11getInstanceEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei"></span><span id="tensorrt_llm::runtime::MemoryCounters::bytesToString__SizeType.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ad3e1b845f41a4245556557b1393881f9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">bytesToString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bytes</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8SizeTypei" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei"></span><span id="tensorrt_llm::runtime::MemoryCounters::bytesToString__DiffType.i"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1ae644945f3b01297e95a0f1724b5b4d8a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">bytesToString</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bytes</span></span>, <span class="kt"><span class="pre">int</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">precision</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters13bytesToStringE8DiffTypei" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mGpuE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters4mGpuE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters4mGpuE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mGpu__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a641ecc1e58aa222f4a7e269aa589e70f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpu</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mGpuE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mCpuE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters4mCpuE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters4mCpuE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mCpu__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a27771c9783292c51de1b8a59aca33d71"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCpu</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters4mCpuE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters7mPinnedE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters7mPinnedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters7mPinnedE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mPinned__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a7eaaa821b0f7cbf8f1a2264e5f1b1c25"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8SizeTypeE" title="tensorrt_llm::runtime::MemoryCounters::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPinned</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters7mPinnedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mGpuDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a091075678e2a77da992d619ccc27ee69"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpuDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mGpuDiffE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mCpuDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a55b97586df47d83ece390da5b9da7ace"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCpuDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8mCpuDiffE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mPinnedDiff__DiffType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a3ac02c1fc74dd309e7680be2c27ecbf0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters8DiffTypeE" title="tensorrt_llm::runtime::MemoryCounters::DiffType"><span class="n"><span class="pre">DiffType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPinnedDiff</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters11mPinnedDiffE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14MemoryCounters9mInstanceE">
<span id="_CPPv3N12tensorrt_llm7runtime14MemoryCounters9mInstanceE"></span><span id="_CPPv2N12tensorrt_llm7runtime14MemoryCounters9mInstanceE"></span><span id="tensorrt_llm::runtime::MemoryCounters::mInstance__MemoryCounters"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1MemoryCounters_1a6de4abe7564f08c88fedabcff29ffeca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">thread_local</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCountersE" title="tensorrt_llm::runtime::MemoryCounters"><span class="n"><span class="pre">MemoryCounters</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInstance</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14MemoryCounters9mInstanceE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="samplingconfig-h">
<h2>samplingConfig.h<a class="headerlink" href="#samplingconfig-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfigE"></span><span id="tensorrt_llm::runtime::SamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType"></span><span id="tensorrt_llm::runtime::SamplingConfig::SamplingConfig__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a1b84a281c4413d5a4df922e9074d85ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig14SamplingConfigE8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9beamWidthE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9beamWidthE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9beamWidthE"></span><span id="tensorrt_llm::runtime::SamplingConfig::beamWidth__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a871887c0857fa88f45fadcd659c20824"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9beamWidthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig11temperatureE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig11temperatureE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig11temperatureE"></span><span id="tensorrt_llm::runtime::SamplingConfig::temperature__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1aa0234e25caafb8f7e2540e635354f1a7"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">temperature</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig11temperatureE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9minLengthE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9minLengthE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9minLengthE"></span><span id="tensorrt_llm::runtime::SamplingConfig::minLength__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1aacc380bc9c6ff46fd2edd112b1eac9c0"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">minLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9minLengthE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::repetitionPenalty__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ab267d53af1ff989a2cda73069dfd42fa"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">repetitionPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig17repetitionPenaltyE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::presencePenalty__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1af40f62527a6a93da70def3daafdc8001"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">presencePenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig15presencePenaltyE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topKE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig4topKE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig4topKE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topK__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ab1588865e1c6e97bfebfe12b7ee34e37"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topK</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topKE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topPE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig4topPE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig4topPE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topP__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a6d960ae9fb60fa44c616cf4b16a6977d"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topP</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig4topPE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig10randomSeedE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig10randomSeedE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig10randomSeedE"></span><span id="tensorrt_llm::runtime::SamplingConfig::randomSeed__OptVec:unsigned-l-l:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a6468682988b1992a5be330aaadc9641e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">unsigned</span></span><span class="w"> </span><span class="kt"><span class="pre">long</span></span><span class="w"> </span><span class="kt"><span class="pre">long</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">randomSeed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig10randomSeedE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9topPDecayE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9topPDecayE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9topPDecayE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPDecay__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ad25e0aec9100bf05d100885677dfaa03"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPDecay</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9topPDecayE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig7topPMinE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig7topPMinE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig7topPMinE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPMin__OptVec:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a2f772d5c0be4d9fe80fdf0e142a731d5"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPMin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig7topPMinE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE"></span><span id="tensorrt_llm::runtime::SamplingConfig::topPResetIds__OptVec:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1ae2a3d686956933a6938925309496a33e"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec"><span class="n"><span class="pre">OptVec</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">topPResetIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig12topPResetIdsE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE"></span><span id="tensorrt_llm::runtime::SamplingConfig::beamSearchDiversityRate__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1aa97844dc62b1cf9eac23d606697a575c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">beamSearchDiversityRate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig23beamSearchDiversityRateE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE"></span><span id="tensorrt_llm::runtime::SamplingConfig::lengthPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a902a829073ae4bc0745374e3f5d07a13"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="tensorrt_llm::runtime::SamplingConfig::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lengthPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig13lengthPenaltyE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE">
<span id="_CPPv3N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE"></span><span id="_CPPv2N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a78311bb676349c17d54fee63f3d54396"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FloatType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime14SamplingConfig9FloatTypeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE">
<span id="_CPPv3I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE"></span><span id="_CPPv2I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1runtime_1_1SamplingConfig_1a82ed7012baf2949351e80937329b530a"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OptVec</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="tensorrt_llm::runtime::SamplingConfig::OptVec::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm7runtime14SamplingConfig6OptVecE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="tllmlogger-h">
<h2>tllmLogger.h<a class="headerlink" href="#tllmlogger-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLoggerE">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLoggerE"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLoggerE"></span><span id="tensorrt_llm::runtime::TllmLogger"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TllmLogger</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><span class="n"><span class="pre">ILogger</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLoggerE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE"></span><span id="tensorrt_llm::runtime::TllmLogger::log__Severity.nvinfer1::AsciiCharCP"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1a052c775ee14bc0d741d26d28c5b3f311"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">log</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">severity</span></span>, <a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">AsciiChar</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">msg</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="k"><span class="pre">override</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger3logE8SeverityPKN8nvinfer19AsciiCharE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger8getLevelEv">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger8getLevelEv"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger8getLevelEv"></span><span id="tensorrt_llm::runtime::TllmLogger::getLevel"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1ae6b565ac5ee12cfcd305c0f2c0bd4b1e"></span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLevel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8getLevelEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity">
<span id="_CPPv3N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity"></span><span id="_CPPv2N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity"></span><span id="tensorrt_llm::runtime::TllmLogger::setLevel__Severity"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1TllmLogger_1a02ca7ebe0eec266f8b6ab4b66e9f0275"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLevel</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">Severity</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">level</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime10TllmLogger8setLevelE8Severity" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="worldconfig-h">
<h2>worldConfig.h<a class="headerlink" href="#worldconfig-h" title="Permalink to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfigE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfigE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfigE"></span><span id="tensorrt_llm::runtime::WorldConfig"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">WorldConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="Permalink to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::WorldConfig::WorldConfig__SizeType.SizeType.SizeType.SizeTypeCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a24619a7cd8eab9da4347680e248b922a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">WorldConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">rank</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11WorldConfigE8SizeType8SizeType8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig7getSizeEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab015227c42d7122c0e33f22e03ac8228"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getSizeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getTensorParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a389ba0bc53cd7fe0259bdb2124ee156f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig20getTensorParallelismEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isTensorParallelCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1af3627cbcb19d1519338fea2e43335633"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTensorParallel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig16isTensorParallelEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelismCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab90953f5e50d6fdef2d3c76ad8295f98"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelism</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig22getPipelineParallelismEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isPipelineParallelCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a37ec8744e7dc7ff6211c319a33ecc628"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isPipelineParallel</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig18isPipelineParallelEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig7getRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig7getRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab99195015c992ba7a735c7880cc62bb6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig7getRankEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getGpusPerNodeCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a32f320679d97d1a278e84bb21af89494"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpusPerNode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig14getGpusPerNodeEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getDeviceCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a35f8f78f02108160d180c0b70f78b414"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig9getDeviceEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a619d3cbd60a9cad3d0232844e8693739"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig23getPipelineParallelRankEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getTensorParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab00da06440a9b4f3305d9811dcd7b61b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTensorParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig21getTensorParallelRankEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isFirstPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a6a5a2512cce31560ae828d7fd1968c22"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFirstPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig27isFirstPipelineParallelRankEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv"></span><span id="tensorrt_llm::runtime::WorldConfig::isLastPipelineParallelRankCCE"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a7e9cc05c32f9a1edf82d4017e34a3bef"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLastPipelineParallelRank</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig26isLastPipelineParallelRankEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv">
<span id="_CPPv3NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv"></span><span id="_CPPv2NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv"></span><span id="tensorrt_llm::runtime::WorldConfig::getPipelineParallelGroupC"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a79368880ccca35154c16cbf409c97430"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPipelineParallelGroup</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm7runtime11WorldConfig24getPipelineParallelGroupEv" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType"></span><span id="tensorrt_llm::runtime::WorldConfig::validConfig__nvinfer1::ILoggerR.SizeType.SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab34b3cafb821617e539a8d736c09e3f5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">validConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logger</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig11validConfigERN8nvinfer17ILoggerE8SizeType8SizeType" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::WorldConfig::mpi__nvinfer1::ILoggerR.SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ad8ec600840b45582dccbc01eb7926ff0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv48nvinfer1" title="nvinfer1"><span class="n"><span class="pre">nvinfer1</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ILogger</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logger</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiERN8nvinfer17ILoggerE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE"></span><span id="tensorrt_llm::runtime::WorldConfig::mpi__SizeType.std::optional:SizeType:.std::optional:SizeType:"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1ab0651f2698fe41426584b3ab69529460"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfigE" title="tensorrt_llm::runtime::WorldConfig"><span class="n"><span class="pre">WorldConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gpusPerNode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensorParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pipelineParallelism</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig3mpiE8SizeTypeNSt8optionalI8SizeTypeEENSt8optionalI8SizeTypeEE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE"></span><span id="tensorrt_llm::runtime::WorldConfig::kDefaultGpusPerNode__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a2d68f68654533c7b0b9be92dae183082"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultGpusPerNode</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig19kDefaultGpusPerNodeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE"></span><span id="tensorrt_llm::runtime::WorldConfig::mTensorParallelism__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a60d3f95841a56216582ff4d1749841db"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensorParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig18mTensorParallelismE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE"></span><span id="tensorrt_llm::runtime::WorldConfig::mPipelineParallelism__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a455c07380f4ab9f5d299de40011d03be"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPipelineParallelism</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig20mPipelineParallelismE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig5mRankE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig5mRankE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig5mRankE"></span><span id="tensorrt_llm::runtime::WorldConfig::mRank__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a834b8b05fadab2839ad94c0389385330"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRank</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig5mRankE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE">
<span id="_CPPv3N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE"></span><span id="_CPPv2N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE"></span><span id="tensorrt_llm::runtime::WorldConfig::mGpusPerNode__SizeType"></span><span class="target" id="classtensorrt__llm_1_1runtime_1_1WorldConfig_1a3c6d093540443a20a3204d0129d8be79"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtime8SizeTypeE" title="tensorrt_llm::runtime::SizeType"><span class="n"><span class="pre">SizeType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpusPerNode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtime11WorldConfig12mGpusPerNodeE" title="Permalink to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="../python-api/tensorrt_llm.runtime.html" class="btn btn-neutral float-left" title="Runtime" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2023, NVidia.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink