mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-31 16:21:07 +08:00
128a351bdc
For VSWA scheme, we do not want `kv_cache_cnonfig.max_token` to control and cap the maximum memory of a block pool because block pool size are not identical amongst different window sizes. This MR omits the effect of `kv_cache_config.max_tokens` under `kvCacheManager.cpp` to allow the setting of block pool size to rely on the window size to share ratio and the total gpu memory analyzed and fed to the kv cache manager. Only skipping for VSWA scheme, no extra coverage was added. Signed-off-by: eopXD <yuehtingc@nvidia.com> |
||
|---|---|---|
| .. | ||
| cmake | ||
| include/tensorrt_llm | ||
| kernels | ||
| micro_benchmarks | ||
| tensorrt_llm | ||
| tests | ||
| CMakeLists.txt | ||
| conandata.yml | ||
| conanfile.py | ||
| libnuma_conan.py | ||