Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multiprocessing Connection import error on OSX #8711

Merged
merged 2 commits into from
May 4, 2020
Merged

Fix multiprocessing Connection import error on OSX #8711

merged 2 commits into from
May 4, 2020

Conversation

jhtimmins
Copy link
Contributor

Currently, dag_processing.py imports the multiprocessing library, and uses multiprocessing.connection.Connection to declare a Connection type for the DagFileProcessorManager class. This causes an import error on OS X.

This happens because the multiprocessing package does not expose the .connectionmodule by default. To access this behavior, you need to call multiprocessing.allow_connection_pickling() first.

Alternatively, instead of importing multiprocessing calling multiprocessing.connection.Connection in your code, you can import the Connection class directly:

from multiprocessing.connection import Connection

This PR makes that update.Presumably, this distinction matters bc of weird rules in Python's import code.


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label May 4, 2020
@ashb
Copy link
Member

ashb commented May 4, 2020

This only affects Py3.8+ right?

@dimberman dimberman self-requested a review May 4, 2020 21:56
@jhtimmins
Copy link
Contributor Author

@ashb No, it comes up in (at least) 3.7.7 as well.

Copy link
Member

@mik-laj mik-laj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This problem also occurs on Linux.
https://github.com/apache/airflow/runs/644247542

@kaxil
Copy link
Member

kaxil commented May 4, 2020

Makes me wonder, what changed !!

@kaxil
Copy link
Member

kaxil commented May 4, 2020

Most likely was introduced in Python 3.7.7

@ashb ashb merged commit 6600e47 into apache:master May 4, 2020
@ashb ashb added this to the Airflow 1.10.11 milestone May 4, 2020
@jhtimmins jhtimmins deleted the import-multiprocessing-connection branch May 4, 2020 22:49
@kaxil kaxil modified the milestones: Airflow 1.10.12, Airflow 2.0.0 Aug 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants