Bug: Memory usage exceeds expectations

[cdboer]

gitlab2prov used ~30GB of memory when run in local testing for a project of ~500 commits and roughly 20 MB of files contained within the repository. The tool should never use that much memory for a project of that size. This issue impacts the scalability of the tool and should be investigated.

Possible reasons:

• git repository is cloned into memory
• computed diffs are stored in memory even when no longer needed
• in memory repository holds on to data longer than necessary

[cdboer]

memray could help to find the root cause.

[cdboer]

May have found the culprit in prov/model.py. Each of the functions concerned with different models has the same default argument pattern.

def deletion(
    ...,
    graph: ProvDocument = graph_factory(),
):
    ...

The graph_factory call creates an empty provenance graph when the function is defined. The same graph is used every time the function is called. Python's default arguments are evaluated when the function is defined and are stored in the __defaults__ attribute of the function. The default arguments are not evaluated every time the function is called. This led to the graphs defined in functions such as deletion or addition to store not just one deletion or addition model at a time but to store all deletions and additions ever recorded for a singe run. This resulted in quite some memory bloat.

[cdboer]

With the bloated graphs sitting in __defaults__

def f(..., graph = graph_factory()):
    ...

Without them sitting in __defaults__

def f(..., graph = None):
    if graph is None:
        graph = graph_factory()

[pablogsal]

👋 Hi @cdboer ,

I am one of the authors of memray. We are collecting success stories here. If you have a minute, do you mind leaving a short message on how memray help with this issue? Knowing how we managed to help will help us track trends and target areas for improvement, prioritize new features and development and identify potential bugs or areas of confusion.

Thanks a lot for your consideration and for helping us improve the profiler :)