是否有办法以内存换显存？

[ovo-Tim]

GeForce RTX 3060 Mobile 6G 只能跑 1.3B 模型，7B 会爆显存，有没有办法挽救一下？
听说有共享 GPU内存 但是 Linux 下似乎没有，求教。

Traceback (most recent call last):
  File "/home/tim/llama/Chinese-LLaMA-Alpaca-2/scripts/inference/inference_hf.py", line 243, in <module>
    generation_output = model.generate(
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/transformers/generation/utils.py", line 1588, in generate
    return self.sample(
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/transformers/generation/utils.py", line 2642, in sample
    outputs = self(
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 824, in forward
    logits = self.lm_head(hidden_states)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/accelerate/hooks.py", line 159, in new_forward
    args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/accelerate/hooks.py", line 286, in pre_forward
    set_module_tensor_to_device(
  File "/home/tim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 317, in set_module_tensor_to_device
    new_value = value.to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 432.00 MiB (GPU 0; 5.80 GiB total capacity; 4.37 GiB already allocated; 75.44 MiB free; 4.79 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[ovo-Tim]

实测 llama.cpp 可跑

[ymcui]

上面的朋友已经回答了，可以使用量化框架，例如llama.cpp。Q4量化的7B模型不到4G左右。

[greatiliad]

好像windows下面可以划一半内存做共享显存，pc机ddr4插满128G，玩玩推理应该也凑合了吧

[smart4654154]

共享 GPU内存 但是 Linux 下似乎没有，请问现在有了吗？