'Partial' Equivalent

[kalzoo]

Question

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.5.1
            pydantic compiled: True
                 install path: /.../venv/lib/python3.7/site-packages/pydantic
               python version: 3.7.4 (default, Sep  7 2019, 18:27:02)  [Clang 10.0.1 (clang-1001.0.46.4)]
                     platform: Darwin-19.4.0-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

Hey Samuel & team,

First of all, huge fan of Pydantic, it makes working in Python so much better. I've been through the docs and many of the issues several times, and until this point I've found all the answers I was looking for. If I missed one in this case - sorry in advance!

Looking to be able to transform a model at runtime to another with the same fields but where all are Optional, akin to Typescript's Partial (hence the issue title). Something similar is discussed in part in at least one FastAPI issue, but since the answer there is basically "duplicate your model" and since this is inherently a pydantic thing, I thought I'd ask here.

By way of a little bit of hacking on model internals, I've been able to get the functionality I need for a relatively trivial use case:

from __future__ import annotations

from pydantic import BaseModel, create_model
from typing import Optional

class BaseSchemaModel(BaseModel):
    # ...

    @classmethod
    def to_partial(cls) -> BaseSchemaModel:
        return get_partial_model(cls)


def get_partial_model(model: BaseModel) -> BaseModel:
    """
    Return a model similar to the original, but where all fields are Optional.

    Note: this minimal implementation means that many Pydantic features will be discarded
    (such as alias). This is not by design and could stand to improve.
    """
    new_fields = {
        name: (Optional[model.__annotations__.get(name)], model.__fields__[name].default or None)
        for name in model.__fields__
    }
    return create_model("GeneratedPartialModel", **new_fields, __config__=model.__config__)

verified with a simple test:

import pytest

from pydantic import ValidationError
from typing import Optional

from app.utilities.pydantic import BaseSchemaModel


def test_partial_model():
    class TestModel(BaseSchemaModel):
        a: int
        b: int = 2
        c: str
        d: Optional[str]

    with pytest.raises(ValidationError):
        TestModel()

    model = TestModel(a=1, c='hello')
    assert model.dict() == {'a': 1, 'b': 2, 'c': 'hello', 'd': None}

    PartialTestModel = TestModel.to_partial()

    partial_model = PartialTestModel()
    assert partial_model.dict() == {'a': None, 'b': 2, 'c': None, 'd': None}

However this doesn't support much of Pydantic's functionality (aliases, for a start). In the interest of a better issue report and possibly a PR, I tried a couple other things, but in the time I allocated did not get them to work:

from pydantic import BaseModel, create_model
from pydantic.fields import ModelField

class Original(BaseModel):
  a: int
  b: str
  c: Optional[str]

class PartialModel(BaseModel):
  a: Optional[int]
  b: Optional[str]
  c: Optional[str]

def copy_model_field(field: ModelField, **kwargs) -> ModelField:
    params = (
      'name',
      'type_',
      'class_validators',
      'model_config',
      'default',
      'default_factory',
      'required',
      'alias',
      'field_info'
    )
    return ModelField(**{param: kwargs.get(param, getattr(field, param)) for param in params})

# Doesn't work - ModelField not acceptable in place of FieldInfo
GeneratedPartialModel = create_model('GeneratedPartialModel', **{name: copy_model_field(field, required=False) for name, field in Original.__fields__.items()})

# Doesn't work - field_info doesn't contain all the necessary information
GeneratedPartialModel = create_model('GeneratedPartialModel', **{name: copy_model_field(field, required=False).field_info for name, field in Original.__fields__.items()})

# This works for my use case - but without aliases and probably without some other functionality as well
new_fields = {name: (Optional[Original.__annotations__.get(name)], Original.__fields__[name].default or None) for name in Original.__fields__}
GeneratedPartialModel = create_model('GeneratedPartialModel', **new_fields)

Would be happy to put in a PR for this, if

a. it doesn't exist already
b. it would be useful
c. I knew where that would best fit - on ModelMetaclass? BaseModel? A utility function to_partial?

Thanks!

[StephenBrown2]

Generally I would make the Partial model a subclass of the Original:

from typing import Optional
from pydantic import BaseModel

class Original(BaseModel):
  a: int
  b: str
  c: Optional[str]

class PartialModel(Original):
  a: Optional[int]
  b: Optional[str]
  c: Optional[str]

which retains the validation and everything else about the original, but makes all the fields it defines Optional.

In : Original(a=1, b="two")
Out: Original(a=1, b='two', c=None)

In : PartialModel(b="three")
Out: PartialModel(a=None, b='three', c=None)

Sinc you already have some optional fields in your original, you don't need to redefine them, though it does help with explicitness:

In : class PartialModel(Original):
...:     a: Optional[int]
...:     b: Optional[str]
...:

In : PartialModel(b="four")
Out: PartialModel(a=None, b='four', c=None)

[kalzoo]

Thanks @StephenBrown2 - yeah, that’s what was suggested in the linked FastAPI issue and is what I’d consider the simplest approach. Simplicity is a plus but it has two important caveats:

• code duplication and vulnerability to drift
• not available at runtime, which is what I’m looking for in this case.

[StephenBrown2]

I would agree, though:

• code duplication and vulnerability to drift
  • If you define your models next to each other, it will be more difficult to drift since you'd see the changes, though they would have to be made in two places, explicit is better than implicit.
• not available at runtime, which is what I’m looking for in this case.
  • I'm not sure what you mean by "at runtime"? The model wouldn't change regardless if you defined it beforehand with the Base model or if you generated it from the Base model, would it? Or maybe I misunderstand.

Even with that said, I can see the benefit of a generated Partial model, for update requests for example. I also don't know the best way to go about that, though, so I'll defer to @samuelcolvin for his thoughts.

[lsorber]

Hi @kalzoo, we're using a different method that does not require code duplication:

First define your Partial model without Optionals, but with a default value equal to a missing sentinel (#1761). That should allow you to create objects where the supplied fields must validate, and where the omitted fields are equal to the sentinel value.

Then, create a Full model by subclassing the Partial model without redefining the fields, and adding a Config class that sets validate_all to True so that even omitted fields are validated.

[mortezaPRK]

hey
I use this mixin to avoid deduplication:

class Clonable(BaseModel):

    @classmethod
    def partial(cls):
        return cls.clone(to_optional='__all__')

    @classmethod
    def clone(
        cls,
        *,
        fields: Set[str] = None,
        exclude: Set[str] = None,
        to_optional: Union[Literal['__all__'], Set[str], Dict[str, Any]] = None
    ) -> 'Clonable':
        if fields is None:
            fields = set(cls.__fields__.keys())

        if exclude is None:
            exclude = set()

        if to_optional == '__all__':
            opt = {f: None for f in fields}
            opt.update(cls.__field_defaults__)
        elif isinstance(to_optional, set):
            opt = {f: None for f in to_optional}
            opt.update(cls.__field_defaults__)
        else:
            opt = cls.__field_defaults__.copy()
            opt.update(to_optional or {})

        model = create_model(
            cls.__name__,
            __base__=Clonable,
            **{
                field: (cls.__annotations__[field], opt.get(field, ...))
                for field in fields - exclude
            }
        )
        model.__name__ += str(id(model))
        return model

[dashavoo]

I needed this too. I have an implementation, but I think this is something that belongs in pydantic itself, and for that some more polish is needed. Does any maintainer have time to have a look and advise? Perhaps @dmontagu as it involves generics? The relevant commit is here: 4cbe5b0

I was going to create a Pull Request, but apparently I'm supposed to open an issue to discuss first, and this issue already exists. If this is welcome (once polished up), I think opening a Pull Request to discuss the details makes sense. Is that what is expected?

[samuelcolvin]

Thanks for using pydantic. 🙏

As part of a migration to using discussions and cleanup old issues, I'm closing all open issues with the "question" label. 💭 🆘 🚁

I hope you've now found an answer to your question. If you haven't, feel free to start a 👉 question discussion.

[samuelcolvin]

Humm, just seen this while reviewing #2245.

I've had this problem too, my solution is to use validate_model directly, see #932.

That way you get errors returned to you and can decide what to do with them. You could even then use construct() to create a model from your values.

The clone() solutions suggested above by @mortezaPRK should also work (I haven't looked through the code there).

You could easily build a utility function for this, either shipped with pydantic or standalone.

The main problem is that type hints and knowledge about the model are no longer valid. You thing you have an instance of (for example) User but actually email is None or raises an AttributeError.

That's one of the advantages of using my validate_model solution above, because you get back a dict, you're less tempted to pretend it's a full model.

---

Overall, I see the problem here, but I'm not yet sure I know what the best solution is.

[cyruskarsan]

Would love if a skip_fields option was provided which would do the same thing as validate_model. Would be a lot cleaner if I could pass it to my BaseModel class
Alternatively, passing the argument to the instance of my BaseModel class would be great as well
MyBaseModel(**the_model).dict(skip_invalid=True)