Skip to content

Add Python object reference tracking #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

pablogsal
Copy link
Member

This feature adds the capability to track Python objects throughout
their lifecycle in memray. The implementation introduces a reference
tracking system that records PyObject creation and destruction events,
storing them as TrackedObject records in the memray capture file with
associated stack traces. The Tracker class now accepts a
reference_tracking flag to enable this functionality and provides a
get_surviving_objects() method to retrieve live objects at the end of
tracking.

Object tracking works across threads and supports native, Python, and
hybrid stack traces, providing insights into Python object
lifetimes. This enhancement extends memray beyond memory allocation
tracking to give developers a powerful tool for understanding object
lifecycles and identifying potential reference leaks.

With this addition, memray can now correlate memory allocations with the
Python objects that trigger them, enabling more precise debugging of
memory usage patterns. The feature is particularly useful for
identifying objects that persist longer than expected or detecting
unexpected object creation patterns. By tracking both memory allocations
and object references, memray provides a more complete picture of a
program's memory behavior, helping developers optimize memory usage and
resolve leaks more effectively.

This implementation leverages the new Python reference tracing API that
was introduced in CPython 3.13.

Signed-off-by: Pablo Galindo pablogsal@gmail.com

Issue number of the reported bug or feature request: #

Describe your changes
A clear and concise description of the changes you have made.

Testing performed
Describe the testing you have performed to ensure that the bug has been addressed, or that the new feature works as planned.

Additional context
Add any other context about your contribution here.

@pablogsal pablogsal force-pushed the pyrefs2 branch 7 times, most recently from 32d2317 to d748e4d Compare April 14, 2025 09:10
@codecov-commenter
Copy link

codecov-commenter commented Apr 14, 2025

Codecov Report

Attention: Patch coverage is 19.15423% with 325 lines in your changes missing coverage. Please review.

Project coverage is 90.46%. Comparing base (864fc2d) to head (a03e9de).

Files with missing lines Patch % Lines
tests/integration/test_object_tracking.py 28.76% 161 Missing ⚠️
src/memray/_memray/record_reader.cpp 0.00% 62 Missing ⚠️
src/memray/_memray/tracking_api.cpp 21.42% 44 Missing ⚠️
src/memray/_memray/records.cpp 0.00% 23 Missing ⚠️
src/memray/_memray/record_writer.cpp 0.00% 15 Missing ⚠️
src/memray/_memray/tracking_api.h 0.00% 13 Missing ⚠️
src/memray/_memray/socket_reader_thread.cpp 0.00% 3 Missing ⚠️
src/memray/_memray/compat.h 0.00% 2 Missing ⚠️
src/memray/_memray/hooks.cpp 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #752      +/-   ##
==========================================
- Coverage   92.97%   90.46%   -2.52%     
==========================================
  Files          95       96       +1     
  Lines       11687    12086     +399     
  Branches      407      421      +14     
==========================================
+ Hits        10866    10933      +67     
- Misses        821     1153     +332     
Flag Coverage Δ
cpp 90.46% <19.15%> (-2.52%) ⬇️
python_and_cython 90.46% <19.15%> (-2.52%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pablogsal
Copy link
Member Author

Tests require 3.13.3 at least because before there is a bug that causes some objects to be identified wrongly. cibuildwheel only has this when pypa/cibuildwheel#2359 passes

@pablogsal pablogsal force-pushed the pyrefs2 branch 4 times, most recently from d5bbf63 to 6b081eb Compare April 14, 2025 13:10
This feature adds the capability to track Python objects throughout
their lifecycle in memray. The implementation introduces a reference
tracking system that records PyObject creation and destruction events,
storing them as TrackedObject records in the memray capture file with
associated stack traces. The Tracker class now accepts a
reference_tracking flag to enable this functionality and provides a
get_surviving_objects() method to retrieve live objects at the end of
tracking.

Object tracking works across threads and supports native, Python, and
hybrid stack traces, providing insights into Python object
lifetimes. This enhancement extends memray beyond memory allocation
tracking to give developers a powerful tool for understanding object
lifecycles and identifying potential reference leaks.

With this addition, memray can now correlate memory allocations with the
Python objects that trigger them, enabling more precise debugging of
memory usage patterns. The feature is particularly useful for
identifying objects that persist longer than expected or detecting
unexpected object creation patterns. By tracking both memory allocations
and object references, memray provides a more complete picture of a
program's memory behavior, helping developers optimize memory usage and
resolve leaks more effectively.

This implementation leverages the new Python reference tracing API that
was introduced in CPython 3.13.

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
@joerick
Copy link

joerick commented Apr 26, 2025

I just released cibuildwheel with 3.13.3 support!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants