How can I limit the usage of cuda memory to avoid CUDA out of memory.

[toolgood]

System.Runtime.InteropServices.ExternalException:“CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 23.99 GiB total capacity; 21.46 GiB already allocated; 0 bytes free; 23.18 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Exception raised from malloc at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\c10\cuda\CUDACachingAllocator.cpp:704 (most recent call first):
00007FFB7D5CAD0200007FFB7D5CACA0 c10.dll!c10::Error::Error [ @ ]
00007FFB7D59524500007FFB7D5951D0 c10.dll!c10::OutOfMemoryError::OutOfMemoryError [ @ ]
00007FFB7D63DC8400007FFB7D63C1D0 c10_cuda.dll!c10::cuda::CUDACachingAllocator::init [ @ ]
00007FFB7D63DD0700007FFB7D63C1D0 c10_cuda.dll!c10::cuda::CUDACachingAllocator::init [ @ ]
00007FFB7D6216F0 c10_cuda.dll! [ @ ]
00007FFAD36B16F100007FFAD36B1620 torch_cpu.dll!at::detail::empty_generic [ @ ]
00007FFB7DA3DD9200007FFB7DA3DCA0 torch_cuda_cpp.dll!at::detail::empty_cuda [ @ ]
00007FFB7DA3DBC200007FFB7DA3DA80 torch_cuda_cpp.dll!at::detail::empty_cuda [ @ ]
000001F9CA76722F000001F9CA740370 torch_cuda_cu.dll!at::cuda::set__symint [ @ ]
00007FFAD36DFF2D00007FFAD36DFA10 torch_cpu.dll!at::TensorIteratorBase::fast_set_up [ @ ]
00007FFAD36DBF6700007FFAD36DBEE0 torch_cpu.dll!at::TensorIteratorBase::build [ @ ]
00007FFAD36DC58000007FFAD36DC490 torch_cpu.dll!at::TensorIteratorBase::build_borrowing_binary_op [ @ ]
000001F9CA8386F9000001F9CA7E32E0 torch_cuda_cu.dll!at::cuda::where_outf [ @ ]
000001F9CA72863F000001F9CA6E6010 torch_cuda_cu.dll!at::cuda::bucketize_outf [ @ ]
00007FFAD3F4AA0800007FFAD3F4A980 torch_cpu.dll!at::_ops::mul_Tensor::redispatch [ @ ]
00007FFAD53FF71500007FFAD53B6BD0 torch_cpu.dll!torch::autograd::GraphRoot::apply [ @ ]
00007FFAD53C9C3A00007FFAD53B6BD0 torch_cpu.dll!torch::autograd::GraphRoot::apply [ @ ]
00007FFAD3EDC24A00007FFAD3EDC0D0 torch_cpu.dll!at::_ops::mul_Tensor::call [ @ ]
00007FFAD36762C400007FFAD36762B0 torch_cpu.dll!at::Tensor::mul [ @ ]
00007FFB68008A3700007FFB680089C0 LibTorchSharp.DLL!THSTensor_mul [ @ ]
00007FFB093A0633 ! [ @ ]
”

[toolgood]

I don't know why the memory of the graphics card can work normally when I run python code. After I convert it to C # code, the memory of the graphics card is not enough.

python code: https://github.com/thuml/Autoformer

csharp code: https://github.com/toolgood/OtherFiles/tree/master/ToolGood.SoarSky.StockFormer

[toolgood]

run python code,

args : --is_training 1 --root_path ./dataset/ETT-small/ --data_path ETTh1.csv --model_id ETTh1_96_96 --model Autoformer --data ETTh1 --features M --seq_len 96 --label_len 48 --pred_len 96 --e_layers 2 --d_layers 1 --factor 3 --enc_in 7 --dec_in 7 --c_out 7 --des 'Exp' --itr 1

[toolgood]

run csharp code

[NiklasGustafsson]

The first step is to try technique 3 outlined in this article: https://github.com/dotnet/TorchSharp/wiki/Memory-Management.

I would read the whole article, but it's technique 3 that is going to be the most convenient.

Start by putting a dispose scope in each of your custom modules' forward() functions. Be careful to return the result of the forward by either moving it to an outer scope (the preferred way), or detaching it (but then you have to manually dispose of it).

public override Tensor forward(Tensor input)
{
    using var _ = torch.NewDisposeScope();
    [...]
    var l41 = ...;
    return relu.forward(l41).MoveToOuterDisposeScope();
}

Then, surround your train and test logic with a dispose scope, preferrably so that temporaries are disposed once every batch, but definitely once every epoch. See CIFAR10 Train and CIFAR10 Test for examples on how to do this.

Within these two placements of dispose scopes, you should have very stable memory management, as stable as Python's reference counting. If that still isn't enough, then your only remaining recourse is to play with the batch size.

[toolgood]

@NiklasGustafsson thank you