Enable the cumem (CUDA VMM) allocator by default on CUDA and ROCm
platforms so users get stable physical pages for KV cache without
needing to set --enable-cumem-allocator explicitly. This is required
for MNNVL KV transfers.
Changes:
- ModelConfig.enable_cumem_allocator defaults to None, resolved to
True when the cumem C extension is available
- Cached fabric handle probe in csrc/cumem_allocator.cpp uses a real
cuMemCreate probe instead of trusting cuDeviceGetAttribute, with
fallback to POSIX FD when fabric handles aren't available
- Safety-net fallback: if fabric was probed as available but a real
allocation still fails, updates the cache and retries with POSIX FD
- Weight loading bypasses cumem pool when sleep mode is off (cumem is
only needed for KV cache stability in that case)
- Guard against stale engine allocations corrupting the singleton
Co-authored-by: Claude <[email protected]>
Signed-off-by: Tyler Michael Smith <[email protected]>
