Modernize all Python 3.8 annotations #1815

ml-evs · 2023-10-15T11:47:54Z

This PR is complementary to #1745. I will first investigate how easy it is to merge the two...

Add pyupgrade --py39-plus as a pre-commit hook and run it
Use 3.9 as base version of CI/pre-commit to avoid any compatibility issues between tools

.pre-commit-config.yaml

optimade/adapters/base.py

optimade/adapters/structures/adapter.py

optimade/adapters/structures/ase.py

optimade/adapters/structures/cif.py

optimade/server/entry_collections/elasticsearch.py

optimade/server/exception_handlers.py

optimade/server/schemas.py

optimade/validator/utils.py

tests/models/test_utils.py

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

ml-evs · 2023-10-15T13:32:23Z

I have to dash, @CasperWA once the tests pass here I will merge and then force update the big migration PR... I have kept a backup at ml-evs/backup-pydantic-v2-orig that we can always go back to, but otherwise I will review the rebased version next week. Thanks again for your help!

codecov · 2023-10-15T13:36:02Z

Codecov Report

Merging #1815 (ef611f8) into master (cf45488) will decrease coverage by 0.10%.
The diff coverage is 99.07%.

@@            Coverage Diff             @@
##           master    #1815      +/-   ##
==========================================
- Coverage   90.77%   90.68%   -0.10%     
==========================================
  Files          74       74              
  Lines        4629     4616      -13     
==========================================
- Hits         4202     4186      -16     
- Misses        427      430       +3

Flag	Coverage Δ
project	`90.68% <99.07%> (-0.10%)`	⬇️
validator	`90.57% <99.07%> (-0.10%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
optimade/adapters/base.py	`96.96% <100.00%> (ø)`
optimade/adapters/references/adapter.py	`100.00% <100.00%> (ø)`
optimade/adapters/structures/adapter.py	`100.00% <100.00%> (ø)`
optimade/adapters/structures/aiida.py	`100.00% <100.00%> (ø)`
optimade/adapters/structures/ase.py	`98.61% <100.00%> (-0.02%)`	⬇️
optimade/adapters/structures/cif.py	`84.09% <100.00%> (-0.36%)`	⬇️
optimade/adapters/structures/proteindatabank.py	`89.28% <100.00%> (-0.13%)`	⬇️
optimade/adapters/structures/pymatgen.py	`98.50% <100.00%> (ø)`
optimade/adapters/structures/utils.py	`80.88% <100.00%> (+0.14%)`	⬆️
optimade/client/client.py	`81.01% <100.00%> (-0.92%)`	⬇️
... and 34 more

JPBergsma · 2023-10-16T17:30:13Z

I think there are still many more locations where we still import the type definitions.
for example: entry_collections/mongo.py
optimade/client/client.py

Should these also be changed?

ml-evs · 2023-10-16T17:35:16Z

I think there are still many more locations where we still import the type definitions. for example: entry_collections/mongo.py optimade/client/client.py

Should these also be changed?

Do you mean Union, Optional etc? These are still required for Python 3.9 (but not for 3.10). I think the pyupgrade hook dealt with everything we can get away with for now.

JPBergsma · 2023-10-16T19:28:48Z

Looks like there was an issue with caching. After a reboot, I no longer get any hits when I search for "Tuple".

* Use Python 3.9 as the 'base' CI version for linting * Update pre-commit hooks * Run `pyupgrade --py39-plus` to upgrade legacy annotations * Add `--exit-non-zero-on-fix` for ruff Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com> * Use f-string over format Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com> --------- Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

* Addded meta field to entry model * Added validator for meta field. * Update pyyaml version in requirements.txt * Update requirements.txt * Update requirements.txt * Remove test structures to get an idea of what triggers yaml import error on github. * Readding meta field to test_good_structures. * Readding meta field to test_structures.json. * Added handling None for property_metadata to validator + small correctionds. * Added test for validator per entry meta field. * Added test for presence metadata field in test_structures.py. * Remove metadata fields when the fields that they belong to are not returned. * add extra test for bad prefix. * Test if dependancy conflict causes error. * Revert "Test if dependancy conflict causes error." This reverts commit 6251e48. * Revert "Revert "Test if dependancy conflict causes error."" This reverts commit bf9692a. * correct version httpx. * commenting out validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github.. * Slowly reassembling validator to see if this resolves issue Github.. * Placed yaml import in try except block. * Added more cases to test data and added bugfix for removing associated metadata when a field is excluded. * Removed seemingly unneccesary mypy ignore exception statement. * remove change in version pyyaml in requirements.txt. * correct spelling mistake * moved starts_with_supported_prefix and check_starts_with_supported_prefix functions to BaseresouceMapper. * Expanded docstring check_starts_with_supported_prefix * Expanded docstring check_starts_with_supported_prefix * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Adjusted validators baseed on suggestion Matthew. Co-authored-by: ml-evs ml-evs@users.noreply.github.com * small corrections validators. * small correction in removing unrequested metadata. * Added spaces before values 'exmpl_originates_from _project'. * update openapijson * update openapijson * update openapijson * use lstrip instead of manually removing / one at a time. * Update optimade/models/entries.py update description property_metadata field Co-authored-by: Matthew Evans <7916000+ml-evs@users.noreply.github.com> * Update description per entry meta datafield. * update openapijson. * try to see if moving yaml is still necessary. * Revert "try to see if moving yaml is still necessary." This reverts commit 68f14d3. * Added supported_prefixes field to config.py * Update requirements-client.txt (#1813) removed duplicate requirement * Modernize all Python 3.8 annotations (#1815) * Use Python 3.9 as the 'base' CI version for linting * Update pre-commit hooks * Run `pyupgrade --py39-plus` to upgrade legacy annotations * Add `--exit-non-zero-on-fix` for ruff Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com> * Use f-string over format Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com> --------- Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com> * Update pydantic validators and model configs for 2024 * Remove validation of 'supported prefix' in lieu of just checking leading underscores --------- Co-authored-by: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

ml-evs added 2 commits October 15, 2023 12:40

Use Python 3.9 as the 'base' CI version for linting

23f8150

Update pre-commit hooks

5b83300

ml-evs requested review from CasperWA, JPBergsma and markus1978 as code owners October 15, 2023 11:47

This was referenced Oct 15, 2023

[IGNORE] Mergeability test #1816

Closed

[IGNORE] Mergeability #2 #1817

Closed

ml-evs force-pushed the ml-evs/remove-py38 branch from e921de4 to 791b932 Compare October 15, 2023 12:59

CasperWA requested changes Oct 15, 2023

View reviewed changes

Run pyupgrade --py39-plus to upgrade legacy annotations

1f0da3e

ml-evs force-pushed the ml-evs/remove-py38 branch from 791b932 to 1f0da3e Compare October 15, 2023 13:26

ml-evs and others added 2 commits October 15, 2023 14:26

Add --exit-non-zero-on-fix for ruff

af658dd

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

Use f-string over format

ef611f8

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

ml-evs merged commit 8510ffa into master Oct 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Modernize all Python 3.8 annotations #1815

Modernize all Python 3.8 annotations #1815

Uh oh!

ml-evs commented Oct 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ml-evs commented Oct 15, 2023

Uh oh!

codecov bot commented Oct 15, 2023

Uh oh!

JPBergsma commented Oct 16, 2023

Uh oh!

ml-evs commented Oct 16, 2023

Uh oh!

JPBergsma commented Oct 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Modernize all Python 3.8 annotations #1815

Modernize all Python 3.8 annotations #1815

Uh oh!

Conversation

ml-evs commented Oct 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ml-evs commented Oct 15, 2023

Uh oh!

codecov bot commented Oct 15, 2023

Codecov Report

Uh oh!

JPBergsma commented Oct 16, 2023

Uh oh!

ml-evs commented Oct 16, 2023

Uh oh!

JPBergsma commented Oct 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants