Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check for ref counting bugs in debug mode caused by immortal objects. #94851

Closed
kumaraditya303 opened this issue Jul 14, 2022 · 10 comments
Closed

Check for ref counting bugs in debug mode caused by immortal objects. #94851

kumaraditya303 opened this issue Jul 14, 2022 · 10 comments
Assignees
@kumaraditya303
Labels
3.12 bugs and security fixes type-bug An unexpected behavior, bug, or error

Comments

@kumaraditya303
Copy link
Contributor

kumaraditya303 commented Jul 14, 2022 •
edited by bedevere-bot
Loading

With #90699, the identifiers are statically allocated and are immortal. This makes it easy to make reference counting mistakes as they are not detected and cause negative ref count in _Py_RefTotal.

On my machine the reference count is negative because of missing incref on &_Py_STR(empty):

@kumaraditya303 ➜ /workspaces/cpython (main) $ ./python -I -X showrefcount -c pass
[-1 refs, 0 blocks]

PR #94850 fixes this issue.

To make it easy to discover reference counting issue, I propose to after each runtime finalization check that all the static allocated immortal objects have ref count of 999999999 otherwise _PyObject_Dump can be used to output the object and abort the process in debug mode and this will help prevent these kinds of issues of "unstable" ref count.

cc @ericsnowcurrently

Linked PRs
@kumaraditya303 kumaraditya303 added type-bug An unexpected behavior, bug, or error 3.12 bugs and security fixes labels Jul 14, 2022
@kumaraditya303 kumaraditya303 self-assigned this Jul 14, 2022
@corona10
Copy link
Member

corona10 commented Jul 14, 2022 •
edited
Loading

Hmm FYI on macOS latest branch: 6cbb57f

➜  cpython git:(main) ✗ ./python.exe -I -X showrefcount -c pass
[0 refs, 0 blocks]

It's not reproducible but I didn't check it on Linux.

@corona10
Copy link
Member

corona10 commented Jul 14, 2022 •
edited
Loading

Same on Ubuntu 20.04.4 LTS 6cbb57f

corona10@python-dev:~/cpython$ ./python -I -X showrefcount -c pass
[0 refs, 0 blocks]

@kumaraditya303
Copy link
Contributor Author

kumaraditya303 commented Jul 14, 2022 •
edited
Loading

I tested on Ubuntu 20.04.4 LTS with 5.4.0-1074-azure kernel.

I think it is dependent on which code path is executed by site module but it doesn't matter as c functions always returns strong references and never borrowed references.

This was referenced Jul 19, 2022
Merged
Merged
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 20, 2022
@kumaraditya303 @miss-islington
pythonGH-94851: fix immortal objects refcounting in compiler (pythong…
b745b27 
…h-95040)

(cherry picked from commit 7476154)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
corona10 pushed a commit that referenced this issue Jul 20, 2022
@kumaraditya303
GH-94851: fix immortal objects refcounting in compiler (gh-95040)
7476154
@corona10
Copy link
Member

@ericsnowcurrently
IIUC, once the https://peps.python.org/pep-0683/ is landed, we do not have to handle these kinds of reference counting operations for _Py_STR cases. Is it correct?

@kumaraditya303
Copy link
Contributor Author

IIUC, once the https://peps.python.org/pep-0683/ is landed, we do not have to handle these kinds of reference counting operations for _Py_STR cases. Is it correct?

Yes, once immortal objects are implemented then we can take advantage of this work and remove all the reference counting, even automate it once we have all the correct reference counting of immortal objects.

@kumaraditya303
Copy link
Contributor Author

Also this issue would be relevant even after immortal objects as even with immortal objects we still want stable -X showrefcount.

@kumaraditya303
Copy link
Contributor Author

FTR, so far the following PRs have been merged which fixed refcounting on immortal objects:

miss-islington added a commit that referenced this issue Jul 20, 2022
@miss-islington @kumaraditya303
GH-94851: fix immortal objects refcounting in compiler (gh-95040)
16cb8ca 
(cherry picked from commit 7476154)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
@ericsnowcurrently
Copy link
Member

@ericsnowcurrently IIUC, once the https://peps.python.org/pep-0683/ is landed, we do not have to handle these kinds of reference counting operations for _Py_STR cases. Is it correct?

Right.

@ericsnowcurrently
Copy link
Member

I suspect we'll drop this check later. However, it is helpful until then.

miss-islington pushed a commit that referenced this issue Jul 25, 2022
@kumaraditya303
GH-94851: check refcnt of immortal objects after finalization (GH-95001)
73ee5a6
@kumaraditya303
Copy link
Contributor Author

Fixed by #95001

@bedevere-bot bedevere-bot mentioned this issue Mar 14, 2023
Merged
kumaraditya303 added a commit that referenced this issue Mar 14, 2023
@kumaraditya303
GH-94851: check unicode consistency of static strings in debug mode (#…
a703f74 
…102684)
carljm added a commit to carljm/cpython that referenced this issue Mar 14, 2023
@carljm
Merge branch 'main' into inlinecomp2
6c5f269 
* main: (50 commits)
  pythongh-102674: Remove _specialization_stats from Lib/opcode.py (python#102685)
  pythongh-102660: Handle m_copy Specially for the sys and builtins Modules (pythongh-102661)
  pythongh-102354: change python3 to python in docs examples (python#102696)
  pythongh-81057: Add a CI Check for New Unsupported C Global Variables (pythongh-102506)
  pythonGH-94851: check unicode consistency of static strings in debug mode (python#102684)
  pythongh-100315: clarification to `__slots__` docs. (python#102621)
  pythonGH-100227: cleanup initialization of global interned dict (python#102682)
  doc: Remove a duplicate 'versionchanged' in library/asyncio-task (pythongh-102677)
  pythongh-102013: Add PyUnstable_GC_VisitObjects (python#102014)
  pythonGH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (pythonGH-102649)
  pythongh-102627: Replace address pointing toward malicious web page (python#102630)
  pythongh-98831: Use DECREF_INPUTS() more (python#102409)
  pythongh-101659: Avoid Allocation for Shared Exceptions in the _xxsubinterpreters Module (pythongh-102659)
  pythongh-101524: Fix the ChannelID tp_name (pythongh-102655)
  pythongh-102069: Fix `__weakref__` descriptor generation for custom dataclasses (python#102075)
  pythongh-98169 dataclasses.astuple support DefaultDict (python#98170)
  pythongh-102650: Remove duplicate include directives from multiple source files (python#102651)
  pythonGH-100987: Don't cache references to the names and consts array in `_PyEval_EvalFrameDefault`. (python#102640)
  pythongh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct (python#102562)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102631)
  ...
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
@kumaraditya303 @Fidget-Spinner
pythonGH-94851: check unicode consistency of static strings in debug …
c92b445 
…mode (python#102684)
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@kumaraditya303 @warsaw
pythonGH-94851: check unicode consistency of static strings in debug …
bddbc3c 
…mode (python#102684)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kumaraditya303 kumaraditya303

Labels
3.12 bugs and security fixes type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ericsnowcurrently @corona10 @kumaraditya303