Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low recusion limit with Python 3.12 #92

Closed
fumitoh opened this issue Dec 24, 2023 · 1 comment
Closed

Low recusion limit with Python 3.12 #92

fumitoh opened this issue Dec 24, 2023 · 1 comment
Labels

Comments

@fumitoh
Copy link
Owner

fumitoh commented Dec 24, 2023

In Python 3.12, C level recusion limit is introduced:
python/cpython#112215
As a result, the maximum number of formula recusions in modelx is significantly lowered from 100,000+ down to slightly less than 500.

Workaound in modelx may be implemented, but Python core developers have intension to introduce a mechanism that adjust the limit according to the available C stack size.
modelx users should be advised not to upgrade Python to 3.12 at the moment.

@fumitoh
Copy link
Owner Author

fumitoh commented Dec 26, 2023

The modelx library is fundamentally recursion-oriented, which presents a unique challenge: enabling practically unlimited recursion.

This challenge has been a focal point since the library's early development stages.
In Python versions prior to 3.11, Python's recursion mechanism triggered C-level recursion, meaning the recursion depth in Python was constrained by the size of the C-stack.

With Python 3.11, there was a significant change: Pure Python recursion won't induce C-level recursion anymore. This separation appeared to address the issue initially. However, the introduction of a hardcoded limit on C-level recursion in Python 3.12
revealed that certain dunder methods, like __call__ and __getitem__, still induce C-level recursion.

Since modelx heavily relies on cells objects calling other cells through __call__ or __getitem__ in their formulas, the recursion limit for modelx formulas is effectively bound by the C-level recursion limit.
Although Python core developers have plans to increase the hardcoded limit on C-level recursion in future Python releases, it doesn't fully resolve the core issue: the modelx recursion limit remains constrained by the C-stack size, which varies across different platforms. For instance, on Windows, the default C-stack size is smaller and can't be altered after thread initiation. This limitation has been circumvented by initiating a new thread with a larger stack size for formula execution. In contrast, Linux allows dynamic increase of the C-stack size.

To address this issue comprehensively, a backward-incompatible change is proposed:

In the namespace associated with a modelx space, cell names will now be bound to the CellsImpl.call method, bypassing the Cells.__call__ method. This adjustment effectively detaches modelx recursion from C-level recursion, rendering the recursion in modelx virtually limitless on all supported platforms (Windows, MacOS, Linux).

However, this change comes with some backward compatibility issues:

  • The ability to call cells within formulas using the subscription operation [] will be removed. This feature was primarily syntactic sugar, so users can adapt their models to use the call operation () instead.
  • Other methods on Cells, such as match, will not be available anymore. If needed, cells can still be accessed directly using _space.cell_name, where cell_name is the specific name of the cells.

This restructuring aims to future-proof modelx against recursion limits across different future Python versions and operating systems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant