Description
openedon Dec 21, 2020
We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.
Describe the bug
We are trying to update our SDK version from 3.9.1 - preview to the latest preview version. We use the change feed pull model. We noticed when we updated our version to 3.15.2 - preview and deployed, our machines quickly ran out of RAM. Looking at the logs, we got up to about 23gb of RAM in use and then crashed. I was able to reproduce locally and noticed that the RAM on my machine quickly ballooned.
I put together a console app to verify the behavior wasn't anything in our code, the only dependencies in the console app is to Cosmos SDK. In my test, I read off the change feed for 10 minutes and monitored the diagnostics in VS2019. I tested with various versions and took screenshots of the memory usage in the diagnostic session. These tests were against a collection that is about ~300GB
3.15.2 - Preview - Ended up at around 1.8gb for the 10 minutes
3.15.1 - Preview - 900mb in 10 minutes
3.15.0 - Preview - 600mb in 10 minutes
3.14.0 - Preview - 150mb in 10 minutes
3.13.0 - Preview - 150mb in 10 minutes
3.9.1 - Preview - 150mb in 10 minutes
Here is the code for the methods I ran:
3.13 and up
public async Task RunFeed()
{
var connection = "";
var container = new CosmosClient(connection).GetContainer("", "");
var totalDocs = 0;
var options = new ChangeFeedRequestOptions
{
PageSizeHint = 5000
};
using (var iterator = container.GetChangeFeedStreamIterator(ChangeFeedStartFrom.Beginning(), options))
{
var watch = Stopwatch.StartNew();
while (iterator.HasMoreResults && watch.Elapsed < TimeSpan.FromMinutes(10))
{
using (var response = await iterator.ReadNextAsync())
{
if (!response.IsSuccessStatusCode)
{
break;
}
using (var changes = await JsonDocument.ParseAsync(response.Content))
{
var count = changes.RootElement.GetProperty("_count").GetInt32();
Console.WriteLine($"Retrieved {count} documents");
totalDocs += count;
}
}
}
Console.WriteLine($"Found {totalDocs} total documents in {watch.Elapsed.TotalMinutes} minutes");
}
}
3.9.1 - Preview
public async Task RunFeed()
{
var connection = "";
var container = new CosmosClient(connection).GetContainer("", "");
var totalDocs = 0;
var iterator = container.GetChangeFeedStreamIterator(changeFeedRequestOptions: new ChangeFeedRequestOptions
{
StartTime = DateTime.MinValue,
MaxItemCount = 5000
});
var watch = Stopwatch.StartNew();
while (iterator.HasMoreResults && watch.Elapsed < TimeSpan.FromMinutes(10))
{
using (var response = await iterator.ReadNextAsync())
{
using (var changes = await JsonDocument.ParseAsync(response.Content))
{
var count = changes.RootElement.GetProperty("_count").GetInt32();
Console.WriteLine($"Retrieved {count} documents");
totalDocs += count;
}
}
}
Console.WriteLine($"Found {totalDocs} total documents in {watch.Elapsed.TotalMinutes} minutes");
}
Environment summary
SDK Version: 3.15.0 - Preview, 3.15.1 - Preview, 3.15.2 - Preview
.Net Core 3.1
