Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid leaked semaphores #214

Merged
merged 3 commits into from
Jan 29, 2024
Merged

Avoid leaked semaphores #214

merged 3 commits into from
Jan 29, 2024

Conversation

troyraen
Copy link
Contributor

@troyraen troyraen commented Jan 26, 2024

Background

We have been encountering "leaked semaphore" warnings occasionally. It's difficult to track down exactly where this is coming from, but I believe we are experiencing issue python/cpython#90549 which reports a bug in python's multiprocessing stemming from the combined use of global resources and "spawning" new processes*. We have been using "spawn" (over the default "fork") because it is thread-safe, which is required for pd.read_parquet, which was being used by ZTF_get_lightcurve. This PR replaces pd.read_parquet with a pyarrow method and switches multiprocessing to "fork".

*Note: A fix has been backported to python>=3.10. I'm still getting the warning using 3.10, but I can't tell if I'm using a version that includes the backport.

Included Changes

  • Remove calls to mp.set_start_method("spawn").
  • Replace pd.read_parquet(..) with pyarrow.parquet.read_table(..).to_pandas().
  • Replace filesystem handler s3fs with pyarrow.fs. This is not strictly necessary, but it's convenient to just use pyarrow for everything. Since no other modules depend on s3fs, I have also removed it from the explicit requirements for light curves. Please let me know if you would prefer to keep it (maybe @bsipocz).

@bsipocz
Copy link
Member

bsipocz commented Jan 26, 2024

Replace filesystem handler s3fs with pyarrow.fs. This is not strictly necessary, but it's convenient to just use pyarrow for everything. Since no other modules depend on s3fs, I have also removed it from the explicit requirements for light curves. Please let me know if you would prefer to keep it (maybe @bsipocz).

I suspect it will be installed anyway as an astropy/pyvo/astroquery dependency, but not directly rely on it sounds good to me, it's always good to keep the dependency footprint to the minimum.

@bsipocz
Copy link
Member

bsipocz commented Jan 26, 2024

*Note: A fix has been backported to python>=3.10. I'm still getting the warning using 3.10, but I can't tell if I'm using a version that includes the backport.

If we can, it would be good to move onto 3.11 or 3.12, so no need to sort out older warnings.

@troyraen
Copy link
Contributor Author

I've run several tests and haven't seen a leaked semaphore warning. Since the warning did not occur every time, I can't be positive that this fixes it. @jkrick can you run a test or two and see if you get the warning? If you don't, then I think this is ready to merge.

@troyraen troyraen marked this pull request as ready for review January 26, 2024 23:29
@troyraen troyraen requested a review from jkrick January 26, 2024 23:30
Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All things equal, any code changes with negative net lines are good, let alone when they also remove a dependency.

Copy link
Contributor

@jkrick jkrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran it with 1,000 and 10,000 sources on the SMCE and both ran successfully.

@troyraen troyraen merged commit ff3ff7f into main Jan 29, 2024
1 check passed
@troyraen troyraen deleted the raen/fix/leaked-semaphore branch January 29, 2024 18:55
github-actions bot pushed a commit that referenced this pull request Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants