Skip to content

Going from SerializerOptions to a custom Serializer makes it impossible to read the data back #1782

Closed

Description

Describe the bug
I started a project using the following CosmosClientOptions:

new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Gateway,
    SerializerOptions = new CosmosSerializationOptions
    {
        PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
    },
}

But eventually, I needed more control over the serialization of my data so I switched to a custom serializer:

new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Gateway,
    Serializer = new CustomCosmosSerializer()
}

Since then, all queries that require composite indexes fail with missing indexes. It looks like the database thinks that my properties are in PascalCase. Changing my indexes to pascal case and trying the queries again yields empty arrays to my serializer instead of triggering an error. It never gets a chance to deserialize any data.

I suspect that my serializer is not used for metadata queries on the Database and Collection objects and cannot participate in the serialization of the query itself. This means that I lost to ability to use LINQ to request objects with camel case properties. Since I cannot use both SerializerOptions and Serializer at the same time to control that internal serializer, I have no way to work around this issue without rewriting a large chunk of code and adding attributes everywhere, which is what I was trying to avoid by using my own serializer in the first place.

To Reproduce

  • Create a command-line app that connects to CosmosDb by setting the PropertyNameingPolicy to camel case like in the previous snippet.
  • Use the client to create some documents.
  • Create a query using the following form to force the use of a composite index:
container.GetItemLinqQueryable<T>()
    .OrderBy(c => c.Prop1)
    .ThenBy(c => c.Prop2)
  • After validating that it does return some results replace the SerializerOptions property with the second snippet, using the following dummy serializer:
public class DummyJsonCosmosSerializer : CosmosSerializer
    {
        public override T FromStream<T>(Stream stream)
        {
            return default;
        }

        public override Stream ToStream<T>(T input)
        {
            return new MemoryStream();
        }
    }

Expected behavior
The stream passed to the serializer should contain the same set of results as if I was using SerializerOptions. In other words, the use of a custom serializer should not change the outcome of the query and the serializer should see the same data in the stream in both scenarios.

Actual behavior
The internal serializer reverts to pascal case causing queries that depends on custom indexes (like an OrderBy().ThenBy() query) to fail with the following message: BadRequest (400) - The order by query does not have a corresponding composite index that it can be served from. Queries that do not require specialized indexes will simply call the serializer with an empty array all the time.

This means that it is currently impossible to switch between SerializerOptions with camel case and a custom Serializer without dumping the database and recreating it from scratch and converting all custom indexes.

Environment summary
SDK Version: 3.12.0
OS Version (e.g. Windows, Linux, MacOSX) Windows 10 build 1909

Notes
I feel like most of this could be avoided if both properties could be used at the same time. At least, I would still be able to read my data and it would be up to me to ensure that my serializer is compatible with the SerializerOptions. The best case scenario would be to use the custom serializer everywhere, but I'm sure there is a reason why it isn't the case already.

Also, no data is lost in this process. You can always switch back to the old version of the code and get everything working again... As long as no documents have been written with the new serializer before you notice that you can't read anything back. Those documents will probably have to be hand-edited to read them with SerializerOptions again. I haven't tried this scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions