Memory Leak on NpgsqlDataSource with password provider

[NikitaCh]

I've originally opened this issue in npsql repo, but after further research I think the issue is in this package, particularly in the fact that connectionString is being used as a key for cache of DataSource

The issue:

There appears to be a memory leak when using EF + NpgsqlDataSource with password provider.

Versions:
Npgsql.EntityFrameworkCore.PostgreSQL - 9.0.4
Npgsql - 9.0.3

Reproducible example attached - MemoryLeak.zip (includes simple docker-compose with postgres for comfort)

Relevant code breakdown:

AppDbContext.cs

internal class AppDbContext : DbContext
{
    public DbSet<SomeModel> SomeModels { get; set; }

    public DbSet<SomeOtherModel> SomeOtherModels { get; set; }

    public AppDbContext(DbContextOptions<AppDbContext> options) : base(options)
    {
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<SomeModel>(b =>
        {
            b.HasKey(e => e.Id);
            b.OwnsOne(e => e.JsonModel).ToJson();
        });

        modelBuilder.Entity<SomeOtherModel>(b =>
        {
            b.HasKey(e => e.Id);
        });

        base.OnModelCreating(modelBuilder);
    }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseNpgsql(@"User ID=postgres;Server=localhost;Port=5433;Database=postgres_db;Pooling=true;", sqlOptions =>
        {
            sqlOptions.EnableRetryOnFailure(maxRetryCount: 3);
            sqlOptions.ConfigureDataSource(dsBuilder =>
            {
                dsBuilder.ConnectionStringBuilder.Host = "localhost";
                dsBuilder.Name = "postgres_db";
                dsBuilder.UsePasswordProvider(_ => Guid.NewGuid().ToString(), (_, _) => ValueTask.FromResult(Guid.NewGuid().ToString()));
            });
        });
        optionsBuilder.UseSnakeCaseNamingConvention();
        base.OnConfiguring(optionsBuilder);
    }
}

Program.cs

using MemoryLeak;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddDbContext<AppDbContext>();

var app = builder.Build();

for(var i = 0; i < 1_000_000; ++i)
{
    using var scope = app.Services.CreateScope();
    var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    await dbContext.Database.CanConnectAsync();

    if (i % 100 == 0)
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        Console.WriteLine($"Iteration: {i}");
        await Task.Delay(3000);
    }
}

From what I can understand, it seems like there is some cache based on ConnectionString, giving the fact that password provider changes the connection string (in this example all the time) - the cache grows indefinitely.

As soon as you change the password provider to return some constant string - everything goes to normal
(For example - dsBuilder.UsePasswordProvider(_ => "constant", (_, _) => ValueTask.FromResult("string"));) :

This is a critical issue, because it is a requirement for a lot of applications to use for example IAM Authorization to connect to RDS - hence use of RDSAuthTokenGenerator for password generation. The application memory grows "indefinitely" in this scenario. The rate of growth seems to be dependent on model size.

Thanks for the help.

[roji]

@NikitaCh I pushed a fix for this - thanks again for reporting.

Can you please give the preview package 9.0.5-ci.20251030T090032 a try, from the stable feed? This should already contain the fix and would help us verify that my fix is correct. If it works, you can safely use this package until 9.0.5 gets released.

[NikitaCh]

@roji, for me to understand, does your fix rely on previous password being rejected by database? Just based on your code changes that is my understanding

[roji]

No, it shouldn't. It simply uses the same connection string and pool as what's used for normal EF operations (before this fix it was created a new one).

[NikitaCh]

@roji thank you! I can confirm that I cannot reproduce it with the same code I used to be able to.

I will play around a bit more though and let you know if I find some other case where this is reproducible, just to make sure 😄 The reason for this is because I definitely knew and confirmed with above code that my services have memory leak because of DbContext health check. But there are other services suffering from similar in intensity memory leak, yet there are no health checks there... Yet I see that your fix specifically targets that (If I understand it correctly), so probably that will lead me to another round of investigation on my side 😕

I have one minor comment though specifically about the fix, the fact that now you are executing SELECT 1 gives an info log, might lead to some confusions/questions coming your way, so might be good to document/note it somewhere, or better yet move to debug?

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (2ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT 1

[NikitaCh]

@roji that didn't take long, but I assume then it is my mis-usage, since I have your attention, would be a miss not to use it 😄

Do I understand correctly, that even if I register DbContext as transient I still have to use scope from DI? Can I have a brief explanation or link where this is described?

So consider below code

using MemoryLeak;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System.Diagnostics;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddDbContext<AppDbContext>(ServiceLifetime.Transient, ServiceLifetime.Singleton);

var app = builder.Build();

Console.WriteLine($"ProcessId: {Process.GetCurrentProcess().Id}");


for (var i = 0; i < 1_000_000; ++i)
{
    //using var scope = app.Services.CreateScope();
    //using var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    using var dbContext = app.Services.GetRequiredService<AppDbContext>();

    await dbContext.Database.CreateExecutionStrategy().ExecuteAsync(async () =>
    {
        return await dbContext.Database.ExecuteSqlRawAsync("SELECT {0}", 1);
    });

    if (i % 100 == 0)
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        Console.WriteLine($"Iteration: {i}");
        await Task.Delay(3000);
    }
}

This has a memory leak.

Uncommenting

    using var scope = app.Services.CreateScope();
    using var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();

And commenting using var dbContext = app.Services.GetRequiredService<AppDbContext>();

Does not have a memory leak.