Potential memory leak if camera is being moved

[rs1659]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues.
• This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

We have a project in our company where we use multiple Zed 2i cameras (currently four, one in each direction for a 360° coverage) mounted to a movable machine and connected to a Jetson AGX Orin in order to detect people coming close to the machine. For this, we have a custom software written in C++ which utilizes the Zed SDK and the provided sample code to interact with the cameras. Now for the tricky part:
We noticed that after some runtime, our software always closed without any reasons given. On further inspection (checking the log of the OS itself), the application process is being killed by the operating system due to running out of system memory and upon further inspection, our application actually managed to fill the 64 GB RAM of the Jetson in about 1-2 days of runtime.

To figure out the problem, we took the setup to the office and let it run here for a while without any noticeable problems even after letting it run for multiple days. The only way to produce the issue seemed to be to move the camera(s) in a way, which brings a person in and out of view of the camera, which resulted in a steady increase in consumed memory.

To provide some further details, I executed the software with Heaptrack attached, which may give some helpful information.

First the heap memory consumption over time:

I moved the camera directly after the startup but as can be noticed from the process memory usage and also in the graph, the memory leak always needs some "spin up" time to get going, then the increase rate seems quite constant, albeit rapid.

The trace also seems quite interesting:

This high memory usage from "fastAMalloc" (and onwards) can only be observed if the camera is moved as described above.

Since the trace here mentions methods which are apparently contained within the sdk of this repo, I think bringing the issue up here is a good starting point.

If you need any further information or have any suggestions, please feel free to ask/tell me, as this is quite an urgent topic for us at the moment.

Steps to Reproduce

1. Run Zed SDK within a C++ application according to sample code
2. Move camera(s) for some time in a way that periodically brings person in and out of view

Expected Result

Application should perform as normal like it is the case if cameras are not being moved.

Actual Result

Application quickly consumes a massive amount of system memory, leading to the process being killed by the OS.

ZED Camera model

ZED2i

Environment

OS: Ubuntu 20.04.6 LTS
PC: NVIDIA Jetson AGX Orin 64 GB

Anything else?

No response

[TanguyHardelin]

Hi @rs1659 ,
Thank you for your reports. Your issue appears to be related to image registration, which is used for detecting loop-closure. If you close the program, you should see the memory usage decrease and return to normal.

As a temporary workaround, you can try disabling and re-enabling the positional tracking every few hours to free up memory. I'm not sure if this approach will be a viable long-term solution for you.

Additionally, it seems from your image that you are using GEN_1 positional tracking. Since this version, we have released two other positional tracking modes: GEN_2 and the latest, GEN_3, which is available in our most recent EA release. These newer modes offer improved accuracy and a lower memory footprint related to loop closure, especially GEN_3.

Please feel free to reach out if you have any questions.

Regards,
Tanguy

[rs1659]

Hello @TanguyHardelin ,

thanks alot for taking the time to investigate and answer. To address your points:

• Using ZED SDK 5.0 - at least in its current state - somehow causes big problems in our application. On the one hand, only between 2-3 of the cameras open (which ones seems random, but never all) with the rest throwing errors due to supposedly "not being detected" even though the cameras and their USB connection are completely fine. The ones that do open also freeze almost instantly after a person is introduced to the scene with no way to recover the camera programmatically. Dropping back down to 4.2.5 fixes these issues. I still switched to GEN_2 for any future endeavours however, even if it is initially way more memory heavy
• Disabling and re-enabling positional tracking actually appears to have a detrimental effect on the issue, since disabling it (via disablePositionalTracking()) doesn't seem to free any memory at all while enabling it again consumes another big chunk of memory (~600 MB it seems). Doing this a few times in quick succession, I can actually fill the entire memory of the Jetson in a few seconds so I assume the issue lies somewhere else? I might add here that after I kill my program, most of the consumed memory is free'd, but not actually 100% of it from what I've gathered so far.

Additionally, you mention the disabling and re-enabling as a "temporary" workaround. Is there anything in the works which addresses this permanently?

If you have any further ideas, I'd be very grateful to hear them but I will continue investigating the root cause in the meantime even though it is extremely time-consuming due to each reproduction consuming upwards of 30 minutes.

[TanguyHardelin]

Hello @rs1659,

I apologize for the delayed response. I attempted to reproduce the issue you're experiencing, but I was unable to do so. Here is the result I obtained when calling disablePositionalTracking() followed by enablePositionalTracking() multiple times:

Could you please share a sample of your code and any associated SVOs? This will help us reproduce the issue on our end and work towards a solution.

Thank you for your cooperation.

Best regards,
Tanguy

[rs1659]

That is indeed rather strange. I will create a sample program which follows the same logic as our actual program and will give you access to it, also - as a side note - we are not using any SVOs within our use case.

As a heads up though this might take a while (>2 weeks) as I currently don't have access to the required hardware but I will see to it ASAP.

[TanguyHardelin]

Hello,

The SVO will help us reproduce the issue on our end so that we can work on fixing it.

I look forward to your response.

Best regards,
Tanguy

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[rs1659]

Status update: I will get access to the hardware soon.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[rs1659]

Hello @TanguyHardelin ,

I am very sorry about the much-longer-than-expected delay but the suppliers for the cameras seemingly have some major delivery issues at the moment. Nonetheless I finally managed to break all of our production code down to a nice, small sample which can be found in this repo.

One thing to note though:
I provided a SVO as requested, however using the SVO doesn't seem to produce the memory leak, only using the live hardware does (this might perhaps be a helpful pointer towards the cause?). I still provided the SVO though, even if it only shows what I mean by my initial description of how the issue can be reproduced.