Could you please reduce the VRAM usage? Or directly run the program on the RAM?

[planb788]

After actual deployment and use, the GPU occupancy easily exceeds 20G, while the video length is only two to three seconds.

[LHY-HongyangLi]

Hi @planb788,
Thank you for your attention to our TAPTR and your kind supplement of environment requirements. Recently I've been busy with another work and can not solve these issues immediately. But I have an idea to reduce the memory usage, and you can have a try by yourself:

In the forward function of TAPTR:

TAPTR/models/dino/taptr.py

Line 353 in 503a333

if not self.training: # eval path

we process all of the images at first and store the resulting feature maps in GPU memory. You can make some changes here to obtain the feature map of one frame only when tracking points on this frame, and delete the feature map after using it.

[LHY-HongyangLi]

Hi, @planb788，
I happened to have the same need this morning. I added a memory-efficient mode to TAPTR. By simply setting this flag to true, the memory requirement can be reduced to only 8GB.

TAPTR/models/dino/taptr.py

Line 116 in 0d3902d

self.memory_efficient_mode = False

[LHY-HongyangLi]

I will close this issue, if you have any questions, feel free to reopen this issue.

[sinkers-lan]

Hi, @LHY-HongyangLi. Thank you for your outstanding work!

I’m curious—does memory_efficient_mode impact the accuracy of your model? I noticed that the evaluation of tapvid_davis_first with memory_efficient_mode is slightly lower than co-tracker.

I look forward to your response!

[LHY-HongyangLi]

It seems that it will indeed hurt the performance, but I have not figure out the reason. 🤕