feat(python): More robust handling of async database calls

[alexander-beedie]

Ref: #15162 (comment)

@stinodego: This brings forward additional async industrialisation that I was working on yesterday, while adding some initial SurrealDB¹ integration (have got a new SurrealDBCursorProxy class working in another branch²).

I haven't been able to reproduce the ResourceWarning locally yet, but this may solve it by explicitly closing any event loops that we know we created in-context (and it doesn't seem to have occurred in these tests, so... maybe? 🤔)

Also improves/fixes interaction with Notebooks/Jupyter (nesting asyncio calls).

Footnotes

1. SurrealDB features: https://surrealdb.com/features ↩

2. SurrealDB + Polars: https://github.com/surrealdb/surrealdb.py/pull/91#issuecomment-2009802667 ↩

[codecov]

Codecov Report

Attention: Patch coverage is 41.17647% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 81.24%. Comparing base (103bb66) to head (9bd5f25).
Report is 23 commits behind head on main.

❗ Current head 9bd5f25 differs from pull request most recent head 638642f. Consider uploading reports for the commit 638642f to get more accurate results

Files	Patch %	Lines
py-polars/polars/io/database/_executor.py	41.17%	9 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15202      +/-   ##
==========================================
- Coverage   81.25%   81.24%   -0.01%     
==========================================
  Files        1354     1354              
  Lines      175390   175406      +16     
  Branches     2518     2520       +2     
==========================================
+ Hits       142505   142506       +1     
- Misses      32404    32417      +13     
- Partials      481      483       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stinodego]

Could you add tests for the cases this is trying to solve? And add nest_asyncio to the requirements-dev.txt.

[alexander-beedie]

Could you add tests for the cases this is trying to solve?

Will see what I can come up with; shouldn't be too hard to setup an outer asyncio loop context and test within that to trigger nesting awareness, hmm 🤔

And add nest_asyncio to the requirements-dev.txt.

Done (had it in the other branch, but missed copying it over 😅)

[alexander-beedie]

Could you add tests for the cases this is trying to solve?

Done; created/committed a test that simulates the event loop conditions found inside Notebooks and other nested async scenarios (such as interop with SurrealDB in an async context manager, which will follow later) 👌

(If you comment out the nest_asyncio.apply() in "_executor.py" you'll see that the test fails with a RuntimeError "asyncio.run() cannot be called from a running event loop", confirming the utility of the new code).

[stinodego]

Thanks Alex, I think we can go ahead with this 👍

I'm only thinking that maybe we can add a warning to the read_database docstring that using async engines is currently considered unstable. That way at least we have some leeway if we choose to go for a different solution in the near future. I'll let you be the judge.

[alexander-beedie]

I'm only thinking that maybe we can add a warning to the read_database docstring that using async engines is current considered unstable. // I'll let you be the judge.

Sounds like a good idea to me; I'll label the support as being considered experimental for now in my imminent follow-up (coming later tonight or first thing tomorrow) that adds SurrealDB support 😎👍