Adopt pydantic-settings with Pydantic V2 #17

hramezani · 2023-04-19T14:09:22Z

Here is the list of changes:

class Config is not valid anymore and has to be replaced with model_config. Example:

    class Settings(BaseSettings):
        apple: str

        class Config:
            env_prefix = 'foobar_'

has to be replaced with:

    class Settings(BaseSettings):
        apple: str

        model_config = ConfigDict(env_prefix='foobar_')

Model.dict() is not available anymore and Model.model_dump() has to be used.
Defining env for a field under class Config is not valid and validation_alias has to be used. Example:

    class Settings(BaseSettings):
        apple: str = ...

        class Config:
            fields = {'apple': {'env': 'BOOM'}}

has to be replaced with:

    class Settings(BaseSettings):
        apple: str = Field(None, validation_alias='BOOM')

Initializing settings class with field name whenever validation_alias is defined does not work anymore. Example:

class SettingsModel(BaseSettings):
    foobar: str = Field(validation_alias='foobar_env')

SettingsModel(foobar='test')  # Raises validation error

Defining env in Field is not valid anymore and validation_alias has to be used. Example:

    class Settings(BaseSettings):
        foobar: str = Field(..., env='foobar_env_name')

has to be replaced with:

    class Settings(BaseSettings):
        foobar: str = Field(validation_alias='foobar_env_name')

customise_sources is not part of class Config anymore and moved to BaseSettings class.

codecov-commenter · 2023-04-19T14:12:12Z

Codecov Report

Patch coverage: 95.65% and project coverage change: -0.81 ⚠️

Comparison is base (0ecac22) 96.31% compared to head (857e057) 95.51%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #17      +/-   ##
==========================================
- Coverage   96.31%   95.51%   -0.81%     
==========================================
  Files           4        5       +1     
  Lines         244      245       +1     
  Branches       60       59       -1     
==========================================
- Hits          235      234       -1     
- Misses          9       10       +1     
- Partials        0        1       +1

Impacted Files	Coverage Δ
pydantic_settings/utils.py	`75.00% <75.00%> (ø)`
pydantic_settings/sources.py	`95.63% <97.33%> (+0.10%)`	⬆️
pydantic_settings/main.py	`100.00% <100.00%> (+1.75%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

dmontagu

LGTM!

I did try to break some stuff related to the implementation of field_is_complex and the only thing I could get to not work was generic dataclasses, which had an easy fix.

I didn't look at the changes to the tests super closely, but from what I could tell they all seemed reasonable. (If you want specific feedback on any of that let me know.)

dmontagu · 2023-04-19T15:18:53Z

pydantic_settings/main.py

-from pydantic.config import BaseConfig, Extra
-from pydantic.fields import ModelField
+from pydantic import ConfigDict, DirectoryPath, FilePath
+from pydantic._internal._utils import deep_update


Are we okay using pydantic internals here? I think it's fine since it's still our package, just want to make sure.

(If we do this, and I guess even if we don't, it might be a good idea to run the tests for pydantic_settings in the CI for pydantic.)

Are we okay using pydantic internals here? I think it's fine since it's still our package, just want to make sure.

Yes, as we maintain Pydantic I think it's fine

(If we do this, and I guess even if we don't, it might be a good idea to run the tests for pydantic_settings in the CI for pydantic.)

Agree

agreed, like we do for fastapi.

dmontagu · 2023-04-19T15:19:35Z

pydantic_settings/main.py

@@ -18,7 +15,7 @@
    SecretsSettingsSource,
 )

-env_file_sentinel = str(object())
+env_file_sentinel: DotenvType = FilePath('')


What motivated this change? Just curious

make mypy happy

dmontagu · 2023-04-19T15:21:15Z

pydantic_settings/main.py

-
-    # populated by the metaclass using the Config class defined above, annotated here to help IDEs only
-    __config__: ClassVar[Type[Config]]
+    model_config = ConfigDict(  # type: ignore[typeddict-item]


Should we make a subclass of ConfigDict that adds more items? I guess that probably doesn't work well.

Probably we'll want to make this work better with the pydantic type hints at some point (annoying that TypedDicts are generally treated as invariant..). But I guess that's outside the scope of a pydantic-settings issue and this approach seems good to me.

Maybe just ignore this comment lol.

yes we should have a custom subclass, in theory it shouldn't break type hinting since we're not changing types, just adding to them, and pydantic will never know the dict has other keys.

SettingsConfigDict added

dmontagu · 2023-04-19T15:23:34Z

pydantic_settings/utils.py

+}
+
+
+def path_type(p: Path) -> str:


Minor thing, I'd suggest renaming this to path_type_label though, to indicate that the returned value is purely meant for display purposes, not for equality checks or anything like that.

samuelcolvin

Otherwise LGTM.

Sorry for the slow response.

samuelcolvin · 2023-04-21T14:55:30Z

pydantic_settings/main.py

-from pydantic.config import BaseConfig, Extra
-from pydantic.fields import ModelField
+from pydantic import ConfigDict, DirectoryPath, FilePath
+from pydantic._internal._utils import deep_update


agreed, like we do for fastapi.

samuelcolvin · 2023-04-21T14:57:19Z

pydantic_settings/main.py

-
-    # populated by the metaclass using the Config class defined above, annotated here to help IDEs only
-    __config__: ClassVar[Type[Config]]
+    model_config = ConfigDict(  # type: ignore[typeddict-item]


yes we should have a custom subclass, in theory it shouldn't break type hinting since we're not changing types, just adding to them, and pydantic will never know the dict has other keys.

samuelcolvin · 2023-04-23T16:36:55Z

pydantic_settings/main.py

+        env_settings: PydanticBaseSettingsSource,
+        dotenv_settings: PydanticBaseSettingsSource,
+        file_secret_settings: PydanticBaseSettingsSource,
+    ) -> Tuple[PydanticBaseSettingsSource, ...]:


not now, but we need to:

add docstrings to this and other methods

rename these methods to all use a consistent prefix, maybe settings_

Implement Protected namespaces pydantic#4915 and use that to make settings_ a protected namespace.

I created an issue for this #19

samuelcolvin · 2023-04-23T16:38:59Z

pydantic_settings/main.py


-from pydantic.config import BaseConfig, Extra
-from pydantic.fields import ModelField
+from pydantic import ConfigDict, DirectoryPath, FilePath


I'm not sure about useing DirectoryPath and FilePath here - I see it's useful semantically to communicate meaning, but it could also be confusing, since they're not enforced at runtime.

I think on balance, it's better not to use these types, probably better to use Path.

DirectoryPath and FilePath replaced by Path

samuelcolvin · 2023-04-23T16:39:32Z

pydantic_settings/main.py

+        file_secret_settings: PydanticBaseSettingsSource,
+    ) -> Tuple[PydanticBaseSettingsSource, ...]:
+        return init_settings, env_settings, dotenv_settings, file_secret_settings
+
    def _build_values(


rename, I guess to _settings_build_values.

hramezani · 2023-04-24T10:13:10Z

Thanks @samuelcolvin for the review. I've addressed all of them.

samuelcolvin

otherwise LGTM.

pydantic_settings/main.py

Adopt pydantic-settings with Pydantic V2

0972a89

dmontagu and others added 4 commits April 19, 2023 09:14

Push tests and fix

e52820d

Push fix

20ac930

upgrade pydantic-core version

d297b1f

Fix lint

ca7db1d

dmontagu approved these changes Apr 19, 2023

View reviewed changes

hramezani force-pushed the pydantic_v2_integration branch from 5bdbff9 to 1eed3ad Compare April 20, 2023 08:58

Add docstring

437093f

hramezani force-pushed the pydantic_v2_integration branch from 1eed3ad to 437093f Compare April 20, 2023 09:01

hramezani added 3 commits April 20, 2023 12:59

Fix types

681acb5

Update to pydantic v2.0a3

eca93d4

Add a test for env file

857e057

hramezani mentioned this pull request Apr 21, 2023

Adopt pydantic-settings docs for Pydantic V2 #18

Merged

samuelcolvin reviewed Apr 23, 2023

View reviewed changes

Address comments

f78cf4d

samuelcolvin reviewed Apr 24, 2023

View reviewed changes

pydantic_settings/main.py Outdated Show resolved Hide resolved

pydantic_settings/main.py Show resolved Hide resolved

rename customise_sources -> settings_customise_sources

cb51270

samuelcolvin enabled auto-merge (squash) April 24, 2023 21:21

samuelcolvin merged commit 1144885 into pydantic:main Apr 24, 2023

hramezani deleted the pydantic_v2_integration branch April 24, 2023 21:24

Uh oh!

Adopt pydantic-settings with Pydantic V2 #17

Adopt pydantic-settings with Pydantic V2 #17

Uh oh!

Conversation

hramezani commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dmontagu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hramezani commented Apr 24, 2023

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hramezani commented Apr 19, 2023 •

edited

Loading

codecov-commenter commented Apr 19, 2023 •

edited

Loading

dmontagu Apr 19, 2023 •

edited

Loading