Expose tracemalloc hook into `_Py_NewReference` for other tracers

[pablogsal]

When new objects are created, the allocation code ends calling into _Py_NewReference (with some exceptions). tracemalloc has a hook into this function to correlate allocations and traces (that do not know anything about objects or references) with newly created objects. This happens here:

cpython/Objects/object.c

Lines 2018 to 2020 in 3d647e7

	if (_Py_tracemalloc_config.tracing) {
	_PyTraceMalloc_NewReference(op);
	}

This allows tracemalloc to be able to return the traceback where objects where allocated, which is great. Unfortunately other debuggers and profilers do not have this capability as there is no way to leverage trace malloc functionality or to hook into object creation.

I propose to expose an API (unclear what layer of the C-API this should be in) where callbacks can be registered to be called in _Py_NewReference and switch tracemalloc to use this API.

Linked PRs

• gh-93502: Add new C-API functions to trace object creation and destruction #115945

[pablogsal]

CC: @vstinner any objection? What do you think?

[vstinner]

I added _PyTraceMalloc_NewReference() to fix an issue with free lists: commit 9e00e80

commit 9e00e80e213ebc37eff89ce72102c1f928ebc133
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 25 13:31:16 2018 +0200

    bpo-35053: Enhance tracemalloc to trace free lists (GH-10063)
    
    tracemalloc now tries to update the traceback when an object is
    reused from a "free list" (optimization for faster object creation,
    used by the builtin list type for example).
    
    Changes:
    
    * Add _PyTraceMalloc_NewReference() function which tries to update
      the Python traceback of a Python object.
    * _Py_NewReference() now calls _PyTraceMalloc_NewReference().
    * Add an unit test.

Issue: #79234

I propose to expose an API (unclear what layer of the C-API this should be in) where callbacks can be registered to be called in _Py_NewReference and switch tracemalloc to use this API.

To have a list of callbacks? As soon as the cost on performance is acceptable, I'm fine with it.

When tracemalloc was also provided as a 3rd party module, I had the same issue: it couldn't handle free lists (hook into _Py_NewReference).

_Py_NewReference() API is not well defined. Sometimes, it's used on a newly allocated memory block. Sometimes, it's used to update an object taken from a free list. Do you think that we should have two different APIs for these two code paths?

I dislike _PyTraceMalloc_NewReference() since it uses an if to the hot path of code creating an object from a free list. I chose correctness over performance.

When new objects are created, the allocation code ends calling into _Py_NewReference (with some exceptions)

Do you have to fix these exceptions to make the code more consistent?

---

By the way, I tried but failed to remove the private _Py_NewReference() function from the C API: #85161 It's used in a few 3rd party projects.

[wangxiang-hz]

@vstinner Sorry for bothering you, I also have a similar requirement, where I want to perform some custom checks when _PyTraceMalloc_NewReference is called. My requirement is as follows:

In our C++ layer, we are adding some Python interfaces for scripting purposes. However, there is always a chance of accidentally missing Py_DECREF, which can lead to memory leaks. For example:

PyObject* name = PyObject_GetAttrString(obj, "obj_name");
Py_RETURN_NONE;

I forget to release name, it will result in a memory leak. If I discover a leak, I can track it down using take_snapshot and compare_to from tracemalloc. However, usually when we discover the leak, a lot of code has been executed, making it difficult to pinpoint the exact location. But keeping tracemalloc running continuously with PyFrame parsing would result in performance loss.

So, I'm wondering if it's possible to do something like registering a PyObj when _PyTraceMalloc_NewReference is called and unregistering it when tracemalloc_free is called. This way, when I'm preparing to exit my program, I will know which PyObj are still held. Once I find them, I can enable tracemalloc at the corresponding locations.

To achieve this, there are two issues that need to be resolved:

1. How to associate the PyObj passed to '_PyTraceMalloc_NewReference' with the ptr passed to tracemalloc_free. Based on my code inspection, it seems feasible.

    PyTypeObject *type = Py_TYPE(op);
    const size_t presize = _PyType_PreHeaderSize(type);
    uintptr_t ptr = (uintptr_t)((char *)op - presize);

2. Currently, unlike tracemalloc_free, there is no easy way to register my own function pointer in _PyTraceMalloc_NewReference. Is there any way to modify this?

Thanks.

[vstinner]

Currently, unlike tracemalloc_free, there is no easy way to register my own function pointer in _PyTraceMalloc_NewReference. Is there any way to modify this?

Can you propose an API for that?

[wangxiang-hz]

Currently, unlike tracemalloc_free, there is no easy way to register my own function pointer in _PyTraceMalloc_NewReference. Is there any way to modify this?

Can you propose an API for that?

With pleasure. I have created a new topic on Discuss, is that okay：
https://discuss.python.org/t/can-we-make-pytracemalloc-newreference-to-support-custom-hooks/41443

Thanks.

[vstinner]

Correct link: https://discuss.python.org/t/can-we-make-pytracemalloc-newreference-to-support-custom-hooks/41443

[wangxiang-hz]

Correct link: https://discuss.python.org/t/can-we-make-pytracemalloc-newreference-to-support-custom-hooks/41443

I'm sorry, I don't know why the link I provided cannot be opened...
I have made some modifications on my branch and it seems feasible:
wangxiang-hz@064be8e

However, I noticed that tracemalloc.h might have forgotten to define extern "C".

[pablogsal]

I have created a PR with an API proposal for this: #115945