Add a SetLoaded method with a lambda callback that allows the state of entities in the collection to be set.

[bart-degreed]

File a bug

After updating EF Core from v6.0.1 to v6.0.2, two of our tests start failing. This looks like a regression in the change tracker, which no longer produces SQL for removing an element from a many-to-many relationship.

The failing tests are:

• Many-to-many relationship: https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/19f655704c91fa89f5b7561e1dd66586d9b521ee/test/JsonApiDotNetCoreTests/IntegrationTests/ReadWrite/Updating/Relationships/RemoveFromToManyRelationshipTests.cs#L239
• Self-referencing many-to-many relationship: https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/19f655704c91fa89f5b7561e1dd66586d9b521ee/test/JsonApiDotNetCoreTests/IntegrationTests/ReadWrite/Updating/Relationships/RemoveFromToManyRelationshipTests.cs#L1001

To understand what's going on: Both tests set up the database, then create an API request to remove an element from an existing many-to-many relationship. Afterwards the database contents is inspected to verify the relationship has been updated.

The logic in the API (system under test) is defined at https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/19f655704c91fa89f5b7561e1dd66586d9b521ee/src/JsonApiDotNetCore/Services/JsonApiResourceService.cs#L460, which delegates to https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/19f655704c91fa89f5b7561e1dd66586d9b521ee/src/JsonApiDotNetCore/Repositories/EntityFrameworkCoreRepository.cs#L450. At a global level, the following happens: The resource service builds an EF Core query that retrieves the parent entity, along with the included subset of related resources to remove. The produced SQL for that is unchanged between EF Core versions:

-- Microsoft.EntityFrameworkCore.Database.Command: Information: Executed DbCommand (2ms) [Parameters=[@__Create_Item1_0='1'], CommandType='Text', CommandTimeout='30']
SELECT w."Id", t."Id", t."FromItemId", t."ToItemId"
FROM "WorkItems" AS w
LEFT JOIN (
    SELECT w1."Id", w0."FromItemId", w0."ToItemId"
    FROM "WorkItemToWorkItem" AS w0
    INNER JOIN "WorkItems" AS w1 ON w0."FromItemId" = w1."Id"
    WHERE w1."Id" = @__Create_Item1_0
) AS t ON w."Id" = t."ToItemId"
WHERE w."Id" = @__Create_Item1_0
ORDER BY w."Id", t."FromItemId", t."ToItemId"

Then the resource repository ensures all returned entities are tracked, marks the to-many relationship as loaded, then removes the requested related resources from the parent entity. In v6.0.1 the change tracker would "see" the removed related entities and produce SQL for that, but in v6.0.2 that no longer happens.

Result from _dbContext.ChangeTracker.DebugView.LongView, after calling _dbContext.ChangeTracker.DetectChanges(); just before SaveChangesAsync in v6.0.1:

WorkItem {Id: 1} Unchanged
  Id: 1 PK
  AssigneeId: <null> FK Unknown
  Description: <null>
  DueAt: <null>
  GroupId: <null> FK Unknown
  ParentId: <null> FK Unknown
  Priority: 'Low'
  Assignee: <null>
  Children: []
  Group: <null>
  Parent: <null>
  Subscribers: []
  RelatedFrom: []
  RelatedTo: []
  Tags: []
WorkItemToWorkItem {FromItemId: 1, ToItemId: 1} Deleted
  FromItemId: 1 PK FK
  ToItemId: 1 PK FK
  FromItem: <null>
  ToItem: <null>

In v6.0.2, this returns:

WorkItem {Id: 1} Unchanged
  Id: 1 PK
  AssigneeId: <null> FK Unknown
  Description: <null>
  DueAt: <null>
  GroupId: <null> FK Unknown
  ParentId: <null> FK Unknown
  Priority: 'Low'
  Assignee: <null>
  Children: []
  Group: <null>
  Parent: <null>
  Subscribers: []
  RelatedFrom: []
  RelatedTo: []
  Tags: []

Include provider and version information

EF Core version: v6.0.2
Database provider: PostgreSQL (tried various 6.0.x versions, no differences observed)
Target framework: .NET 6
Operating system: Windows 10 x64
IDE: Visual Studio Enterprise 2022 v17.0.5

[AndriySvyryd]

This might be caused by the fix to #26779

Please try disabling the fix by calling this on startup:

AppContext.SetSwitch("Microsoft.EntityFrameworkCore.Issue26779", true);

@ajcvickers

[bart-degreed]

Thanks, I can confirm that adding the AppContext switch makes the problem go away.

Do you consider using this switch as a permanent solution or do you intend to fix this breaking change in the next patch release?

[AndriySvyryd]

Do you consider using this switch as a permanent solution or do you intend to fix this breaking change in the next patch release?

We need to investigate further to determine whether this can be fixed in a patch release

[ajcvickers]

@bart-degreed Can you point me to where the model is configured? I would expect the relationship here to be required, but it looks like it is optional.

[bart-degreed]

The model is configured in https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/19f655704c91fa89f5b7561e1dd66586d9b521ee/test/JsonApiDotNetCoreTests/IntegrationTests/ReadWrite/ReadWriteDbContext.cs.

[bart-degreed]

I have created a simplified repro, see below. In reality, our code is much more dynamic. Commenting out the AppContext.SetSwitch line makes both tests fail on EF Core 6.0.2.

#nullable enable

using System.Diagnostics;
using FluentAssertions;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.ChangeTracking;
using Microsoft.Extensions.Logging;

// Bug workaround for https://github.com/dotnet/efcore/issues/27436
AppContext.SetSwitch("Microsoft.EntityFrameworkCore.Issue26779", true);

await CanRemoveSelfFromCyclicManyToManyRelationship();
await CanRemoveFromManyToManyRelationshipWithExtraRemovalsFromResourceDefinition();

async Task CanRemoveSelfFromCyclicManyToManyRelationship()
{
    // Arrange
    WorkItem otherWorkItem = new();

    WorkItem existingWorkItem = new()
    {
        RelatedFrom = new List<WorkItem>
        {
            otherWorkItem
        }
    };

    await using (var dbContext = new AppDbContext())
    {
        await dbContext.Database.EnsureDeletedAsync();
        await dbContext.Database.EnsureCreatedAsync();

        dbContext.WorkItems.Add(existingWorkItem);
        await dbContext.SaveChangesAsync();

        existingWorkItem.RelatedFrom.Add(existingWorkItem);
        await dbContext.SaveChangesAsync();
    }

    int relatedFromIdToRemove = existingWorkItem.Id;

    // Act
    await using (var dbContext = new AppDbContext())
    {
        // Fetch minimal required data (no tracking)
        WorkItem workItemToUpdate = await dbContext.WorkItems
            .Include(workItem => workItem.RelatedFrom)
            .Where(workItem => workItem.Id == existingWorkItem.Id)
            .Select(workItem => new WorkItem
            {
                Id = workItem.Id,
                RelatedFrom = workItem.RelatedFrom
                    .Where(relatedFromWorkItem => relatedFromWorkItem.Id == relatedFromIdToRemove)
                    .Select(relatedFromWorkItem =>
                        new WorkItem
                        {
                            Id = relatedFromWorkItem.Id
                        }).ToList()
            }).FirstAsync();

        // Ensure retrieved data is tracked
        dbContext.GetTrackedOrAttach(workItemToUpdate);
        workItemToUpdate.RelatedFrom = workItemToUpdate.RelatedFrom.Select(workItem => dbContext.GetTrackedOrAttach(workItem)).ToList();
        dbContext.Entry(workItemToUpdate).Collection(workItem => workItem.RelatedFrom).IsLoaded = true;

        // Apply the change
        workItemToUpdate.RelatedFrom = workItemToUpdate.RelatedFrom.Where(workItem => workItem.Id != relatedFromIdToRemove).ToList();

        await dbContext.SaveChangesAsync();
    }

    // Assert
    await using (var dbContext = new AppDbContext())
    {
        WorkItem workItemInDatabase = await dbContext.WorkItems
            .Include(workItem => workItem.RelatedFrom)
            .Include(workItem => workItem.RelatedTo)
            .FirstAsync(workItem => workItem.Id == existingWorkItem.Id);

        workItemInDatabase.RelatedFrom.Should().HaveCount(1);
        workItemInDatabase.RelatedFrom[0].Id.Should().Be(otherWorkItem.Id);

        workItemInDatabase.RelatedTo.Should().BeEmpty();
    }
}

async Task CanRemoveFromManyToManyRelationshipWithExtraRemovalsFromResourceDefinition()
{
    // Arrange
    WorkItem existingWorkItem = new()
    {
        Tags = new List<Tag>
        {
            new(),
            new(),
            new()
        }
    };

    await using (var dbContext = new AppDbContext())
    {
        await dbContext.Database.EnsureDeletedAsync();
        await dbContext.Database.EnsureCreatedAsync();

        dbContext.WorkItems.Add(existingWorkItem);
        await dbContext.SaveChangesAsync();
    }

    int tagIdToRemove = existingWorkItem.Tags[1].Id;
    int extraTagIdToRemove = existingWorkItem.Tags[2].Id;

    // Act
    await using (var dbContext = new AppDbContext())
    {
        // Fetch minimal required data (no tracking)
        WorkItem workItemToUpdate = await dbContext.WorkItems
            .Include(workItem => workItem.Tags)
            .Where(workItem => workItem.Id == existingWorkItem.Id)
            .Select(workItem => new WorkItem
            {
                Id = workItem.Id,
                Tags = workItem.Tags
                    .Where(tag => tag.Id == tagIdToRemove)
                    .Select(tag => new Tag
                    {
                        Id = tag.Id
                    })
                    .ToList()
            })
            .FirstAsync();

        // Ensure retrieved data is tracked, add extra tag to-be-removed.
        dbContext.GetTrackedOrAttach(workItemToUpdate);
        workItemToUpdate.Tags.Add(new Tag { Id = extraTagIdToRemove });
        workItemToUpdate.Tags = workItemToUpdate.Tags.Select(tag => dbContext.GetTrackedOrAttach(tag)).ToList();
        dbContext.Entry(workItemToUpdate).Collection(workItem => workItem.Tags).IsLoaded = true;

        // Apply the change
        int[] tagIdsToRemove = { tagIdToRemove, extraTagIdToRemove };
        workItemToUpdate.Tags = workItemToUpdate.Tags.Where(tag => !tagIdsToRemove.Contains(tag.Id)).ToList();

        await dbContext.SaveChangesAsync();
    }

    // Assert
    await using (var dbContext = new AppDbContext())
    {
        WorkItem workItemInDatabase = await dbContext.WorkItems
            .Include(workItem => workItem.Tags)
            .FirstAsync(workItem => workItem.Id == existingWorkItem.Id);

        workItemInDatabase.Tags.Should().HaveCount(1);
        workItemInDatabase.Tags.Single().Id.Should().Be(existingWorkItem.Tags.ElementAt(0).Id);

        List<Tag> tagsInDatabase = await dbContext.Tags.ToListAsync();
        tagsInDatabase.Should().HaveCount(3);
    }
}

public abstract class Entity
{
    public int Id { get; set; }
}

public sealed class WorkItem : Entity
{
    public IList<Tag> Tags { get; set; } = new List<Tag>();

    public IList<WorkItem> RelatedFrom { get; set; } = new List<WorkItem>();
    public IList<WorkItem> RelatedTo { get; set; } = new List<WorkItem>();
}

public sealed class Tag : Entity
{
    public IList<WorkItem> WorkItems { get; set; } = new List<WorkItem>();
}

public sealed class WorkItemToWorkItem
{
    public WorkItem FromItem { get; set; } = null!;
    public WorkItem ToItem { get; set; } = null!;
}

public sealed class AppDbContext : DbContext
{
    public DbSet<WorkItem> WorkItems => Set<WorkItem>();
    public DbSet<Tag> Tags => Set<Tag>();

    protected override void OnConfiguring(DbContextOptionsBuilder builder)
    {
        builder.UseSqlite("Data Source=sample.db;Pooling=False");
        builder.LogTo(message => Debug.WriteLine(message), LogLevel.Information);
    }

    protected override void OnModelCreating(ModelBuilder builder)
    {
        builder.Entity<WorkItem>()
            .HasMany(workItem => workItem.RelatedFrom)
            .WithMany(workItem => workItem.RelatedTo)
            .UsingEntity<WorkItemToWorkItem>(right => right
                    .HasOne(joinEntity => joinEntity.FromItem)
                    .WithMany(),
                left => left
                    .HasOne(joinEntity => joinEntity.ToItem)
                    .WithMany());
    }
}

public static class DbContextExtensions
{
    /// <summary>
    /// If not already tracked, attaches the specified entity to the change tracker in <see cref="EntityState.Unchanged" /> state.
    /// </summary>
    public static TEntity GetTrackedOrAttach<TEntity>(this DbContext dbContext, TEntity entity)
        where TEntity : Entity
    {
        var trackedEntity = (TEntity?)dbContext.GetTrackedIdentifiable(entity);

        if (trackedEntity == null)
        {
            dbContext.Entry(entity).State = EntityState.Unchanged;
            trackedEntity = entity;
        }

        return trackedEntity;
    }

    /// <summary>
    /// Searches the change tracker for an entity that matches the type and ID of <paramref name="entity" />.
    /// </summary>
    private static object? GetTrackedIdentifiable<TEntity>(this DbContext dbContext, TEntity entity)
        where TEntity : Entity
    {
        Type entityClrType = entity.GetType();
        EntityEntry? entityEntry = dbContext.ChangeTracker.Entries().FirstOrDefault(entry => IsEntity<TEntity>(entry, entityClrType, entity.Id));

        return entityEntry?.Entity;
    }

    private static bool IsEntity<TEntity>(EntityEntry entry, Type entityClrType, int id)
        where TEntity : Entity
    {
        return entry.Entity.GetType() == entityClrType && ((TEntity)entry.Entity).Id == id;
    }
}

Hope this helps to narrow it down.

[ajcvickers]

@bart-degreed The issue here is that the state of join entity remains Added after the code to attach it and paint appropriate state. For example:

WorkItem {Id: 1} Unchanged
  Id: 1 PK
  RelatedFrom: []
  RelatedTo: [{Id: 1}]
  Tags: []
WorkItemToWorkItem {FromItemId: 1, ToItemId: 1} Added
  FromItemId: 1 PK FK
  ToItemId: 1 PK FK
  FromItem: <null>
  ToItem: <null>

and

Tag {Id: 2} Unchanged
  Id: 2 PK
  WorkItems: [{Id: 1}]
Tag {Id: 3} Modified
  Id: 3 PK
  WorkItems: [{Id: 1}]
TagWorkItem (Dictionary<string, object>) {TagsId: 2, WorkItemsId: 1} Unchanged
  TagsId: 2 PK FK
  WorkItemsId: 1 PK FK
TagWorkItem (Dictionary<string, object>) {TagsId: 3, WorkItemsId: 1} Added
  TagsId: 3 PK FK
  WorkItemsId: 1 PK FK
WorkItem {Id: 1} Unchanged
  Id: 1 PK
  RelatedFrom: []
  RelatedTo: []
  Tags: []

This tells EF that the join entity does not exist in the database, and hence trying to delete it would be an error. However, before the bug fixed in #26779, EF was attempting to delete the entity anyway, which in your case worked since even though the state is Added, the entity did actually exist.

The fix is to make sure that the join entity is in the correct state when painting the state of attached disconnected entities. (Assuming, of course, that you actually need to attach disconnected entities rather than using a tracking query.) For example, after setting IsLoaded for the collection:

dbContext.Entry(workItemToUpdate).Collection(workItem => workItem.RelatedFrom).IsLoaded = true;

// Set the state of the join entity
dbContext.Entry(dbContext.Set<WorkItemToWorkItem>().Find(workItemToUpdate.Id, relatedFromIdToRemove)!)
    .State = EntityState.Unchanged;

// Apply the change
workItemToUpdate.RelatedFrom = workItemToUpdate.RelatedFrom.Where(workItem => workItem.Id != relatedFromIdToRemove).ToList();

and

dbContext.Entry(workItemToUpdate).Collection(workItem => workItem.Tags).IsLoaded = true;

// Apply the change
int[] tagIdsToRemove = { tagIdToRemove, extraTagIdToRemove };
// Set the state of join entities...
foreach (var tagId in tagIdsToRemove)
{
    dbContext.Entry(dbContext.Set<Dictionary<string, object>>("TagWorkItem").Find(tagId, workItemToUpdate.Id)!)
        .State = EntityState.Unchanged;
}

workItemToUpdate.Tags = workItemToUpdate.Tags.Where(tag => !tagIdsToRemove.Contains(tag.Id)).ToList();

Note for triage: we should consider making it easier to change the state of join entities such that they are Unchanged, possibly with a new SetLoaded method that would both mark the collection as loaded and mark all contained entities as Unchanged.