Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mypy typing for airflow/models and their tests #20272

Merged
merged 6 commits into from
Dec 14, 2021

Conversation

ashb
Copy link
Member

@ashb ashb commented Dec 13, 2021

The re-ordering of setting attributes in BaseOperator is because
something about that function (throwing the exceptions?) causes mypy
to think that BaseOperator objects could be missing those attributes

Part of #19891


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added the area:API Airflow's REST/HTTP API label Dec 13, 2021
@ashb ashb requested review from uranusjr and potiuk December 13, 2021 22:48
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nits and questions

@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Dec 14, 2021
@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Few statics to fix too.

The re-ordering of setting attributes in BaseOperator is because
_something_ about that function (throwing the exceptions?) causes mypy
to think that BaseOperator objects could be missing those attributes
@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

Few statics to fix too.

Yeah, flake8 was having odd problems and not running for me locally last night (wasn't giving any errors, but just saying it failed. I didn't dig in to it closely as it was late)

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

For reference this is the flake8 error I get:

Run flake8...............................................................................Failed
- hook id: flake8
- exit code: 1

No need to pull the image. Yours and remote cache hashes are the same!

cat: /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7: No such file or directory
must specify at least one container source

ERROR: The previous step completed with error. Please take a look at output above 

cat: /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7: No such file or directory
Container name cannot be empty

ERROR: The previous step completed with error. Please take a look at output above 

Container ID file found, make sure the other container isn't running or delete /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7

ERROR: The previous step completed with error. Please take a look at output above 


No need to pull the image. Yours and remote cache hashes are the same!

Container ID file found, make sure the other container isn't running or delete /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7

ERROR: The previous step completed with error. Please take a look at output above 

Container ID file found, make sure the other container isn't running or delete /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7

ERROR: The previous step completed with error. Please take a look at output above 

Container name cannot be empty

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

MyPy now not reporting anything for models/ buut it looks like MSSQL might be unhapy.

Fix one break one 😞

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Container name cannot be empty

breeze build-image --python 3.7 should fix it - but I will see why it fails in the absence of manifest.

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Hey @ashb - the cat: /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7: No such file or directory might be a side effect of expired GITHUB_TOKEN . Can you please double check this ?

Unfortunately (I raised it to github) if you have expired token, you even can't download public images (!) which you can when you have no GITHUB_TOKEN at all!

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

If that's confirmed I might want to add some protection agains that :)

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

I have no GITHUB_TOKEN in the env, and /home/ash/code/airflow/airflow/manifests does exist and contains a few files.

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

Ah -- but I do have ghcr.io in my ~/.docker/config.json -- let me try removing that

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

No dice.

$ docker logout ghcr.io
...
$ git commit
...
Run flake8...............................................................................Failed
- hook id: flake8
- exit code: 1

cat: /home/ash/code/airflow/airflow/manifests/remote-airflow-manifest-image-3.7: No such file or directory
must specify at least one container source

ERROR: The previous step completed with error. Please take a look at output above 

@ashb
Copy link
Member Author

ashb commented Dec 14, 2021

non-verbose and verbose output of running flake8 here @potiuk https://gist.github.com/ashb/791aa5b98d12942fd871fa4291bb6346

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

AaaaaaH I know! Flake checks run in parallell and remove each other generated file !

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Fix here @ashb #20294

@ashb ashb removed the full tests needed We need to run full set of tests for this PR to merge label Dec 14, 2021
@ashb ashb merged commit d8e190c into apache:main Dec 14, 2021
@ashb ashb deleted the mypy-in-models branch December 14, 2021 18:40
@potiuk potiuk added the mypy Fixing MyPy problems after bumpin MyPy to 0.990 label Dec 15, 2021
@kaxil kaxil added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jan 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) mypy Fixing MyPy problems after bumpin MyPy to 0.990
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants