ObjectStoragePath cannot be deserialized to object

[chxxvi]

Apache Airflow version

3.1.1

If "Other Airflow 2/3 version" selected, which one?

n/a

What happened?

In v3.1.1, if I have a object (pydantic model) which has a field of ObjectStoragePath, passed from a task to another task (xcom), the field will be a string instead of ObjectStoragePath in the 2nd task. There was no problem in v3.1.0.

What you think should happen instead?

deserialize back to ObjectStoragePath instead of string

How to reproduce

n/a

Operating System

AlmaLinux 9.6

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

official helm chart 1.18.0
on-premise native K8s 1.33.5

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[amoghrajesh]

@chxxvi can you please provide min reproducible steps for this one?

[mykola-shyshov]

I'm able to reproduce. Will prepare fix.

================================================================================
TEST 1: ObjectStoragePath standalone serialization
================================================================================
Serialized JSON: {"__classname__": "airflow.sdk.io.path.ObjectStoragePath", "__version__": 1, "__data__": {"path": "s3://my-


================================================================================
TEST 2: ObjectStoragePath in Pydantic model
================================================================================
SERIALIZED JSON: {"__classname__": "__main__.MyModel", "__version__": 1, "__data__": {"name": "test", "path": "s3://my-bucket/my-file.txt"}}

[amoghrajesh]

@mykola-shyshov I do not follow the code snippet, but I will assign this bug to you, go ahead and attempt to fix it. Ping here if you need help

[mykola-shyshov]

Hello. The issue is quite tricky. Due to the recent changes in #51059, all Pydantic model serialization/deserialization is now going through Pydantic dump.
The different cases are explained below:

Serialization Flows Summary

ObjectStoragePath with mode='python' ✅

Serialization:

1. Pydantic serializer calls model.model_dump(mode='python')
2. Returns: {'name': 'test', 'path': ObjectStoragePath(...)} ← Object preserved
3. Airflow recursively serializes the dict
4. Finds ObjectStoragePath has serialize() method
5. Calls ObjectStoragePath.serialize()
6. Returns full type info: {__classname__, __version__, __data__}

Deserialization:

1. Sees __classname__: ObjectStoragePath
2. Calls ObjectStoragePath.deserialize()
3. Reconstructs ObjectStoragePath object ✅

---

ObjectStoragePath with mode='json' ❌ (CURRENT - BROKEN)

Serialization:

1. Pydantic serializer calls model.model_dump(mode='json')
2. Pydantic converts ObjectStoragePath → string
3. Returns: {'name': 'test', 'path': 's3://bucket/file.txt'} ← Lost object!
4. Airflow sees only a string, no type info saved

Deserialization:

1. Calls PathModel.model_validate({'path': 's3://...'})
2. Pydantic expects ObjectStoragePath but receives string
3. No validator to convert string → ObjectStoragePath
4. Field stays as string ❌

Problem: Pydantic strips type info in step 2, Airflow never sees the object.

---

AnyUrl with mode='python' ❌ (ISSUE #56736)

Serialization:

1. Pydantic serializer calls model.model_dump(mode='python')
2. Returns: {'name': 'test', 'url': AnyUrl(...)} ← Object preserved
3. Airflow recursively serializes the dict
4. Checks if AnyUrl has serialize() method → NO
5. Checks if AnyUrl is dataclass → NO
6. Checks if AnyUrl has attrs → NO
7. Raises error: "cannot serialize AnyUrl" ❌

Problem: No Airflow serializer registered for AnyUrl.

---

AnyUrl with mode='json' ✅ (CURRENT - WORKS)

Serialization:

1. Pydantic serializer calls model.model_dump(mode='json')
2. Pydantic converts AnyUrl → string
3. Returns: {'name': 'test', 'url': 'http://example.com/'}
4. Airflow serializes as string

Deserialization:

1. Calls UrlModel.model_validate({'url': 'http://example.com/'})
2. Pydantic expects AnyUrl, receives string
3. Pydantic has built-in validator for AnyUrl
4. Converts string → AnyUrl ✅

Works because: Pydantic knows how to rebuild AnyUrl from string.

---

Summary

Type	Mode	Result	Why
ObjectStoragePath	python	✅	Has serialize() method, Airflow handles it
ObjectStoragePath	json	❌	Pydantic converts to string, type info lost
AnyUrl	python	❌	No Airflow serializer for AnyUrl
AnyUrl	json	✅	Pydantic converts to/from string automatically

[mykola-shyshov]

Documentation says (implemented as in public doc): serializers -> serialize() method -> @DataClass or @attr.define

Airflow out of the box supports three ways of custom serialization. Primitives are returned as is, without additional encoding, e.g. a str remains a str. When it is not a primitive (or iterable thereof) Airflow looks for a registered serializer and deserializer in the namespace of airflow.serialization.serializers. If not found it will look in the class for a serialize() method or in case of deserialization a deserialize(data, version: int) method. Finally, if the class is either decorated with @DataClass or @attr.define it will use the public methods for those decorators.

Code docs: serialize() method -> serializers -> @DataClass or @attr.define

Values that are not of a built-in type are serialized if a serializer is
found for them. The order in which serializers are used is
1. A serialize function provided by the object.
2. A registered serializer in the namespace of airflow.serialization.serializers
3. Annotations from attr or dataclass.

As for me, it would be nice to have a way to override any serialization logic by custom user serializers. That’s probably what I would expect. I think there were reasons to do so.

[mykola-shyshov]

for mode=python user needs to wrap class. case for AnyUrl (pydantic )

class UrlWrapper:
    anyurl: AnyUrl

    def __init__(self, url):
        self.anyurl = AnyUrl(url)

    def serialize(self):
        """Airflow will call this"""
        return {"url": str(self.anyurl)}

    @classmethod
    def deserialize(cls, data, version):
        """Airflow will call this"""
        return cls(data["url"])

class MyModel(BaseModel):
    model_config = {"arbitrary_types_allowed": True}
    name: str
    url: UrlWrapper  # Use wrapper instead of AnyUrl directly

for mode=json user needs to write model serializers and validators. case for ObjectStoragePath

class PathModel(BaseModel):
    model_config = {"arbitrary_types_allowed": True}
    path: ObjectStoragePath

    @field_validator("path", mode="before")
    @classmethod
    def parse_path(cls, v):
        return ObjectStoragePath(v)

    @field_serializer("path")
    def dump_path(self, v: ObjectStoragePath):
        """Convert ObjectStoragePath to string for JSON serialization"""
        return str(v)

[mykola-shyshov]

Tricky question for me — if we want to go with a solution "native" to Airflow, then it should use mode="python" and the serialize methods.
However, for Pydantic it seems more natural to use mode="json", even though that breaks the logic for non-model classes inside models.
Waiting for feedback 😄