feat(parser): add support for Pydantic v2

[leandrodamascena]

Issue number: #2672
Additional issue: #2427

Summary

Changes

After an extensive week of investigation, experimentation, and implementation, I'm excited to open this pull request, which introduces support for Pydantic v2 in the parser feature. I wanted to provide you with an overview of the changes and considerations made during this process:

Support for Pydantic v1 and v2:

• The parser utility has been enhanced to ensure compatibility with both Pydantic v1 and v2.
• Users now have the flexibility to choose their preferred version of Pydantic for parse utility.

Powertools and the support for Pydantic:

• Historically, Powertools has considered Pydantic v1 as the official library for parse utility, given its long-standing compatibility and robustness within our code.
• However, with this update, we are embracing the advancements and enhancements introduced in Pydantic v2, allowing users to leverage the latest features and improvements.
• It's important to highlight that while we have introduced support for Pydantic v2, our official version for Powertools remains Pydantic v1. This decision has been made to prioritize compatibility and provide a seamless experience for our existing users. By maintaining Pydantic v1 as our official library, we ensure that our ecosystem and the applications built with it continue to function as expected without any disruptions.

No-breaking changes were made

NOTE: We haven't detected any breaking changes, but please let us know as soon as possible if you find any.

• Importantly, this update does not introduce any breaking changes to the existing functions within the Powertools framework. The functions that were already in use remain intact and fully functional, ensuring compatibility with the existing codebase.
• However, it's worth noting that with the support for Pydantic v2, some Pydantic functions used in Powertools are now considered deprecated. This deprecation serves as a signal to users that these functions will be removed in a future major release, specifically Powertools v3.

Compatibility table

Pydantic v1	Pydantic v2	V2 deprecation	In use?	Code change required
@validator	@field_validator	✔️	✔️	✔️
@root_validator	@model_validator	✔️	✔️	✔️
.parse_obj()	model_validate()	✔️	✔️
.json()	model_dump_json()	✔️	✔️
.dict()	model_dump()	✔️	✔️
.parse_raw()	model_validate_json()	✔️	✔️

• To ensure a smooth transition and minimize disruptions for our users, we have temporarily suppressed the PydanticDeprecatedSince20 and PydanticDeprecationWarning warnings (related to these functions). This allows existing applications to continue functioning as expected without outputting warnings.
• If needed, you can enable the warnings yourself with something like the code below. Reference: https://docs.python.org/3/library/warnings.html

from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, envelopes
from aws_lambda_powertools.utilities.parser.models import (
    SqsModel,
)

from aws_lambda_powertools import Logger
import pydantic

import warnings
warnings.simplefilter('default')

How to test before we merge this PR

To facilitate the testing of Pydantic v2 before merging this pull request, we have created a separate branch called poc/pydanticv2 where you can find the relevant code changes. Here are the steps to follow in order to test it effectively:

If using sourcecode:
- Clone Powertools repository
- Fetch the poc/pydanticv2 branch
- Check out the branch locally by running the command: git checkout poc/pydanticv2.

If using pip:

Create or modify the requirements.txt file and add the following content:

git+https://github.com/aws-powertools/powertools-lambda-python@poc/pydanticv2#egg=aws_lambda_powertools
pydantic>2

Then install by running: pip install -r requirements.txt.

If using Poetry:

poetry add git+https://github.com/aws-powertools/powertools-lambda-python#poc/pydanticv2
poetry add pydantic>2

Note for documentation

Since the Powertools currently relies on Pydantic v1 as its official dependency, you need to change the aws-lambda-powertools[tracer] or aws-lambda-powertools[all] installation to aws-lambda-powertools or aws-lambda-powertools[validation,tracer,aws-sdk] and add Pydantic v2 as an extra dependency.

Missing in this PR

• Documentation
• Workflow in develop branch to ensure that any changes will work in Pydanticv2
• Community feedback

User experience

Please share what the user experience looks like before and after this change

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[heitorlessa]

hooorraay

[heitorlessa]

Also add a requirements.txt

git+....
pydantic>=2

and poetry command to add this dep for customers to test

[leandrodamascena]

Hi Powertools customers and community at large! We have the PR for Pydanticv2 ready for testing and would love to hear your feedback. Our goal is to merge this PR on July 21st and it would be nice to hear something before then.

Tagging people we know are directly interested in this support: @ran-isenberg @djfurman @half2me @wurstnase @taharah @wickeat @waveFrontSet

[leandrodamascena]

I'm pretty sure you're already catching this warning, though: If I run the same code as a script the warning won't show up.

Hi @waveFrontSet, can you test again?

[waveFrontSet]

Hi @leandrodamascena, updated to commit 1774a1c, still the same behaviour. I'm seeing the same warnings while running pytest, but not while executing it as a script.

.../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    return model.parse_obj(event)
.../lib/python3.10/site-packages/pydantic/main.py:944: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

[leandrodamascena]

Hi @leandrodamascena, updated to commit 1774a1c, still the same behaviour. I'm seeing the same warnings while running pytest, but not while executing it as a script.

.../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    return model.parse_obj(event)
.../lib/python3.10/site-packages/pydantic/main.py:944: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

Hi @waveFrontSet! I could reproduce the problem in my local environment and fixed it. Can you check again?

Thank you.

[waveFrontSet]

Hi @leandrodamascena,
this has done the trick! No warnings when running pytest. Great job. 👍

[rubenfonseca]

Superb work Leandro! The only concern I have is that we are calling "disable_pydantic_v2_warning" in so many different places, and I wonder if we could improve performance by memoizing it so it only runs once, no matter how many times it gets called. What do you think?

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information