Description
openedon Feb 8, 2024
This is a grouping of related issues. Feel free to vote (👍) for this issue to indicate that this is an area that you think we should spend time on, but consider also voting for individual issues for things you consider especially important.
EF9
The following work is a roughly prioritized list of work tentatively planned for the EF9 release. More details of the full scope of Cosmos work over several releases can be found in All work section below.
Important: This plan is not a commitment; it will evolve as we continue to learn throughout the release. Some things not currently planned for EF9 may get pulled in. Some things currently planned for EF9 may get punted out.
The general approach for Cosmos in EF9 is:
- Necessary infrastructure/test work
- Bring the query pipeline up-to-date with relational
- As much as possible in parallel, implement high-value enhancements
- Bring complex types support close to relational level
- Small config items, like RBAC
- High-value smaller query items
- Depending on how this work goes, stretch goals are:
- Better use of partition keys (Will do if query is in good shape in time.)
- Partial updates and CUD batching
Completed
General improvements
- Throw for sync APIs when using Cosmos since they do sync over async #32563
- Update building guide for Cosmos testing #19814
- [Cosmos] Support collections of scalar types #4179
- CosmosDB with RBAC doesn't work #32197
- Cosmos db Hierarchical Partitions in EF Core #28186
- Why can't a CosmosDB partition key be an int when working with EF Core? #28265
- Cosmos: Values passed to
WithPartionKey
should be model values, not store values #33535 - Cosmos: Non-generic Find/FindAsync on DbContext is not translated to ReadItem #33881
- Cosmos DB: Find/FindAsync performs SQL API query when entity has embedded entities #24202
- Cosmos: Add FormattableString FromSql #29177
- Cosmos: add support for pagination #24513
- Cosmos: support vector search #33783
- Indexes are created by default for foreign keys in non-relational databases #34053
- Cosmos: relationship cannot be configured when key is class type #26239
- Cosmos: warn when attempting to save entity with CLR default key value #34180
- Cosmos: Validate our current container mapping strategy #34005
- Cosmos: Inherit partition key definition from parent by convention #34176
- Cosmos: Use 1:1 mapping for .NET
Id
to JSONid
unless explicitly configured otherwise #34179 - Cosmos: Switch the default discriminator property to $type #34269
Query fundamentals
- Cosmos: NullReferenceException when projecting out an owned entity which doesn't own additional entity types #34068
- Align Cosmos query pipeline code with latest relational code #33032
- Use EF8-style primitive collections in the Cosmos provider #25364
- Cosmos: Support queries into primitive collections #25765
- Cosmos: Translate composition over embedded non-primitive collection #16926
- Align Cosmos query pipeline code with latest relational code #33032
- Cosmos: translate Contains over subquery #33905
- Cosmos: translate indexing into collection #25701
- Cosmos: translate Length/Count on collections #25700
- Cosmos: add support for ElementAt(OrDefault) #29497
- Cosmos: Support aggregate operators #16146
- EFCore 3.1 + CosmosDb can't translate any() on owned object #20441
- Cosmos: bad SQL generated when projecting out "Value" #33858
- Cosmos: Support projecting out (primitive) collections #33797
- Cosmos: Implement proper source alias management #33894
- Cosmos: Data corruption due to missing subquery pushdowns (block such scenarios) #34123
Function translations
- Cosmos: Add support for IS_DEFINED() to EF.Functions #17722
- Cosmos: Implement EF.Functions.CoalesceUndefined #33904
- Cosmos: Translate string.Contains, StartsWith, EndsWith with ComparisonType #25250
- Cosmos: translate date/time component members/methods #33975
ReadItem and partition key management
- Cosmos: Detect partition key filters in more queries #20350
- Cosmos: Optimize more queries that could use ReadItem #20693
- Cosmos: We incorrectly execute via ReadItem, ignoring additional operators composed on top #34085
- Cosmos: Extra clauses are removed from the query when extracting hierarchical partition key values #33960
- Cosmos: WithPartitionKey() is ignored when executing query via ReadItem #34102
Infra/internal work
- Cosmos tests sometimes run out of internal service providers #32647
- Cosmos: remove null semantics compensation for InExpression #31063
- Clean up container management in Cosmos #33898
- Cosmos: FeedIterator and ResponseMessage aren't properly disposed #34092
Planned
No more planned work for 9 at this time.
Punted
- Cosmos: Projecting out owned entities retrieves the entire document #34067
- Cosmos: Implement proper entity equality and contains logic for owned (and complex) types #34027
- CosmosDb: Investigate why pooling does not work in Northwind tests #12878
- [Cosmos] How to set a partition key in a embedded entity? #25548
- Cosmos: Advanced type mapping inference (for scalar collections) #34026
- Cosmos: ReadItem should be used for Reload #33893
Stretch
- Cosmos: subquery pushdown #33968
- Add Cosmos support for complex types #31253
- There is a lot of scope here; some of it will be heavily query dependent. We're unlikely to do it all.
- Cosmos: Allow partial updates #13559
- Cosmos: CUD batching #17308
- Question on Cosmos DB database provider owned object querying null #24087
- Cosmos: support toplevel Any() #33854
T-shirt sizes are:
- Small: Days or less
- Medium: Days to a couple of weeks
- Large: Several weeks
- X-Large: Many weeks
All work
These are areas either fundamental to the underlying architecture and implementation, and/or fundamental user experiences that users expect, on one side, from any EF Core provider, and on the other side by any mechanism used to access Cosmos. For example, SelectMany
is a fundamental query pattern that should work on any EF Core provider. Similarly, partition keys are fundamental to working with Cosmos, and hence EF must expose/provide this functionality.
Query fundamentals
At the time of writing (beginning of EF9 work), the Cosmos query pipeline code has fallen behind the relational code by a considerable about. The architecture now employed by relational providers must be ported to Cosmos before any other major changes are made to the query pipeline.
Following this, some fundamental improvements are needed to the query pipeline. Many of these areas have model building, change tracking, update pipeline aspects as well, but they involve a lot of query work.
- Cosmos: Detect partition key filters in more queries #20350 (Medium)
- Cosmos: Add support for IS_DEFINED() to EF.Functions #17722 (Small)
- Cosmos: add support for pagination #24513 (Medium)
- Collections of primitive, complex, and entity types. (Large?)
- Cosmos: remove null semantics compensation for InExpression #31063
--- EF9 cut line ---
- Cosmos: Translate GroupBy #17313 (X-Large?)
- Cosmos: Translate SelectMany #17312 (Medium?)
- Cosmos: Translate subquery in filter condition #17957 (Medium?)
- Cosmos: Support aggregate operators #16146 (Medium?)
- Cosmos: Allow terminating operators after Skip and Take #24571 (Small?)
- Cosmos: support toplevel Any() #33854
- Cosmos: Translate composition over embedded non-primitive collection #16926
Non-query fundamentals
Many of these areas have query aspects, but much of the work can be done independently of query.
- Add Cosmos support for complex types #31253 (X-Large)
- CosmosDB with RBAC doesn't work #32197 (Small)
- Cosmos db Hierarchical Partitions in EF Core #28186 (Small)
- Cosmos: Allow partial updates #13559 (Large?)
- Cosmos: CUD batching #17308 (Large?)
- Cosmos: relationship cannot be configured when key is class type #26239 (Small)
- [Cosmos] How to set a partition key in a embedded entity? #25548 (Small)
--- EF9 cut line ---
- Cosmos: allow to map non-primitive collections to JSON objects #30743 (Large)
- Cosmos: Support composite, hash and range indexes #17303 (Small)
- Cosmos: Honor ValueGenerated #23729 (Medium)
- Cosmos: allow excluding null values from Json document #26981 (Medium)
- Cosmos: Allow to specify consistency level for CUD operations #17309 (Small)
- Ensure all Cosmos client options are exposed/available via EF Core #33034 (Medium)
- Implement interceptors for Cosmos #26478 (Medium)
- Does EFCore support Cosmos DB always encrypt? #28580 (Small)
- Cosmos: Upsert #33201
- Cosmos: support for mapping ReadOnlyMemory and/or similar #34021
Higher-level experiences
--- EF9 cut line ---
- Model versioning and evolution:
- Schema versioning pattern with Cosmos DB #23753
- Cosmos: Data migration #11099
- Cosmos: Support for weakly typed json #20584 (Likely dependency for this work.)
- [Cosmos DB] Support a default value for non-nullable properties #21006 (Another likely dependency for this work.)
- Use shape of documents in the Cosmos database to scaffold a schema #30290
- Data denormalization in Ef Core and Azure Cosmos Db #33727
- Cosmos: Allow database management operations using RBAC #33964
Other work
Query enhancements
--- EF9 cut line ---
- Cosmos: Add Include support #16920
- Cosmos: Selecting complex properties (object or collections mapped as "OwnsOne" and "OwnsMany") makes the query return the entire document #27440
- Cosmos: Allow Many-to-Many relationships using a FK list without any Join Entity #23523
- CosmosDB Provider Support Any(condition) on Nested Collection by mapping to ARRAY_CONTAINS #32626
- Cosmos: translate array projection #26662
- Cosmos: Translate LeftJoin #17314
- Cosmos: Expand checks against null values for nested properties to first check parents for null #17746
- Query: add translation for string.Compare for Cosmos #20910
- The behavior of Distinct in SqlServer and Cosmos is inconsistent #26547
- Cosmos: Add FormattableString FromSql #29177
- Cosmos: remove null semantics compensation for InExpression #31063
- Cosmos: support computed properties #33796
- Cosmos: supporting projecting out arrays directly, without wrapping in an object #33779
Query optimizations
- Cosmos: Improve single-partition queries #17318 (Large)
- Cosmos: Optimize more queries that could use ReadItem #20693 (Large)
--- EF9 cut line ---
- CosmosDb: Optimize into "IN" operator #12635
- Cosmos: Optimize query SQL #13159
- Cosmos: Cache the generated SQL query and evaluate parameter values lazily #17297
- Cosmos: Leverage complete discriminator mapping #20268
- Cosmos: support vector search #33783
Bugs
--- EF9 cut line ---
- Cosmos: Bitwise operators on booleans return incorrect results #13168
- Cosmos: Avoid projecting contained properties #16142
- Cosmos: bad exception for queries with aggregate selecting non-mapped type #20677
- Cosmos: Stop nesting results in an extra JSON object #25527 (What does the SDK do?)
- Cosmos: Issues when loading derived type via base set and vice versa #25886
- CosmosDB Provider with FindAsync + Private Constructor results in Exception #29286
- Cosmos: Owned entity type collection gets wrongly projected out as null when inside an anonymous object projection #31696
- Incorrect naming of calculated values in projections #32021
- Cosmos: Average truncates results #26378
- Cosmos: removing a value from an owned collection leaves the change tracker in a bad state #33687
- UseCosmos with managed identity is not working correct #33328
- Cosmos: some interceptor tests are flaky on Linux #33600
Nice-to-have
--- EF9 cut line ---
- Cosmos: Add API to map every entity type hierarchy to a separate container #13317
- Cosmos: Support UDFs #15338
- Cosmos: Support unique keys #17305
- Cosmos: Entity splitting #17315
- Cosmos: Spatial types, functions and spatial indexes #17317
- Cosmos: CLI tools #17321
- Cosmos: Translate non-persisted property in query when possible #17670
- Cosmos: Support trigger execution #19944
- Cosmos: Improve materialization exceptions #22374
- Cosmos: Expose retry policy options #23203
- Support CanConnect (for health checks) on Cosmos #23501
- Cosmos: Warn for unsupported features #23690
- Cosmos DB ETag If-None-Match query support #24313
- Cosmos: Allow passing Stream as parameter in raw SQL #25526
- Cosmos: Average truncates results #26378
- Support Jagged and Multidimensional Arrays (Cosmos DB) #26824
- EnsureCreated fails with CosmosDB if Scale not set #26885
- No longer possible to initialize CosmosDB client #28018
- Cosmos: add support for ElementAt(OrDefault) #29497
- Cosmos: allow customization of the container's indexing policy #32517
- Cosmos: improve collection shaper perf #16924
- CosmosManyToManyJoinEntityTypeConvention should remove the properties on the join entity type corresponding to the old partition key #33652
- Cosmos NativeAOT support #33909
Testing and internal quality
--- EF9 cut line ---
- Cosmos provider makes extensive use of InternalEntityEntry #16707
- Cosmos: Review tests throwing client-evaluation exceptions #17246
- Cosmos: Add tests for keyless entities #17316
- Cosmos: Implement all specification tests #17333
- Run Cosmos tests on Helix #19973
- Use testcontainer for testing Cosmos DB #33189
- Cosmos: Make Value projection first class in SelectExpression #21348
- Cosmos: Improve query tree during translation #23538
- Cosmos: Evaluate the data types in server side to see need of convert node #25120
- Update internal Cosmos provider code for NRTs #26558
- NorthwindAggregateOperatorsQueryCosmosTest.Distinct_Scalar fails on CI #27688
- Cosmos: Add perf tests #14546
Currently blocked/infeasible
--- EF9 cut line ---