Controlling Continuous GPU Memory Allocation via Environment Variable

[thomasbergersen]

Can there be an environment variable (e.g., export LLAMA_CUDA_FORCE_MAX_MEM=8192) to enforce contiguous GPU memory allocation? On Ada 6000 , if llama-server is launched first, VLLM throws an OOM error, but if VLLM is initialized first, the OOM does not occur.