TdsParserStateObject.TryReadPlpBytes: System.ArgumentException: Offset and length were out of bounds

[frankbuckley]

Since updating to 6.1.0-preview2.25178.5, I am seeing occassional exceptions with the following stack trace:

System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
   at System.Buffer.BlockCopy(Array src, Int32 srcOffset, Array dst, Int32 dstOffset, Int32 count)
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]& buff, Int32 offset, Int32 len, Int32& totalBytesRead, Boolean canContinue, Boolean writeDataSizeToSnapshot, Boolean compatibilityMode)
   at Microsoft.Data.SqlClient.TdsParser.TryReadSqlStringValue(SqlBuffer value, Byte type, Int32 length, Encoding encoding, Boolean isPlp, TdsParserStateObject stateObj)
   at Microsoft.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName, SqlCommand command)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn, Boolean forStreaming)
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(Task task, Object state)
   at Microsoft.Data.SqlClient.SqlDataReader.ContinueAsyncCall[T](Task task, SqlDataReaderBaseAsyncCallContext`1 context)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at DSE.Content.Domain.Commands.GetResourceCommandHandler.DoExecuteAsync(GetResourceCommand command, ValueResultBuilder`1 builder, CancellationToken cancellationToken) in /_/src/content/src/DSE.Content.Domain/Commands/GetResourceCommandHandler.cs:line 81
   at DSE.Framework.DomainModel.Commands.AsyncResultCommandHandler`3.DoExecuteAsync(TCommand command, CancellationToken cancellationToken) in /_/src/framework/src/DSE.Framework.DomainModel/Commands/AsyncResultCommandHandler.cs:line 97

Expected behavior

Read succeeds.

Further technical details

Microsoft.Data.SqlClient version: 6.1.0-preview2.25178.5
.NET target: .NET 9.0.6
SQL Server version: Azure SQL Server
Operating system: mcr.microsoft.com/dotnet/aspnet:9.0.6-noble-amd64

Additional context

Not seen with 6.0.2

Only seen with occasional results. Generally works. So far, I cannot spot a material difference between what works and what doesn't. (Any suggestions on what to look for?)

[Wraith2]

I need a reproduction. It's possible that it's caused by the changes I've made for perf but without being able to investigate I can't be certain or fix it.

[frankbuckley]

I need a reproduction. It's possible that it's caused by the changes I've made for perf but without being able to investigate I can't be certain or fix it.

Well, after a few hours I was just about to give up, but then...

The following pretty reliably reproduces the exception for me, running against Azure SQL database with 6.1.0-preview2.25178.5.

It seems to require quite large rows (though with varying sizes) with connections closed in between.

using Microsoft.Data.SqlClient;

const string connectionString = """
    Server=tcp:TEST_DB_SERVER.database.windows.net,1433;
    Initial Catalog=TEST_DB;
    Persist Security Info=False;
    User ID=USER_ID;
    Password=PASSWORD;
    MultipleActiveResultSets=False;
    Encrypt=True;
    TrustServerCertificate=False;
    Connection Timeout=30;
    """;


var conn0 = new SqlConnection(connectionString);

await conn0.OpenAsync();

Console.WriteLine("Creating table...");

var createTableCommand = conn0.CreateCommand();

createTableCommand.CommandText = """
    if object_id('content', 'U') is not null
        drop table content;

    create table content
    (
        id   bigint identity(1,1) not null primary key,
        text varchar(max) not null,
        text2 varchar(max) not null,
        text3 varchar(max) not null
    );
    """;

await createTableCommand.ExecuteNonQueryAsync();

Console.WriteLine("Inserting rows...");

for (int i = 0; i < 1000; i++)
{
    var insertCommand = conn0.CreateCommand();

    insertCommand.CommandText = $"""
    insert into content (text, text2, text3)
    values 
    (
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}'
    ),
    (
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}'
    ),
    (
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}'
    ),
    (
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}'
    ),
    (
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}',
        '{string.Join(' ', Enumerable.Range(0, Random.Shared.Next(1000)).Select(i => Random.Shared.Next()))}'
    )
    """;

    await insertCommand.ExecuteNonQueryAsync();
}

await conn0.CloseAsync();
await conn0.DisposeAsync();

var conn1 = new SqlConnection(connectionString);

var selectIdsCommand = conn1.CreateCommand();

selectIdsCommand.CommandText = """
    select          id
    from            content
    """;

await conn1.OpenAsync();

Console.WriteLine("Selecting IDs...");

using var idsReader = selectIdsCommand.ExecuteReader();

List<long> ids = [];

while (await idsReader.ReadAsync())
{
    ids.Add(idsReader.GetInt64(0));
}

await conn1.CloseAsync();
await conn1.DisposeAsync();

Console.WriteLine("Running test...");

foreach (var id in ids)
{
    var conn2 = new SqlConnection(connectionString);

    var selectCommand = conn2.CreateCommand();

    selectCommand.CommandText = $"""
    select          id,
                    text,
                    text2,
                    text3
    from            content
    where           id = {id}
    """;

    await conn2.OpenAsync();

    using var reader = await selectCommand.ExecuteReaderAsync();

    while (await reader.ReadAsync())
    {
        Console.WriteLine($"id: {reader[0]}");
        string text = reader.GetString(1);
        string text2 = reader.GetString(2);
        string text3 = reader.GetString(3);
    }

    await conn2.CloseAsync();
    await conn2.DisposeAsync();
}

I get a slightly different stack trace with this than that reported above:

   at System.Buffer.BlockCopy(Array src, Int32 srcOffset, Array dst, Int32 dstOffset, Int32 count)
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadStringWithEncoding(Int32 length, Encoding encoding, Boolean isPlp, String& value)
   at Microsoft.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName, SqlCommand command)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn, Boolean forStreaming)
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(Task task, Object state)
   at Microsoft.Data.SqlClient.SqlDataReader.ContinueAsyncCall[T](Task task, SqlDataReaderBaseAsyncCallContext`1 context)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Program.<<Main>$>d__0.MoveNext() in V:\dev\frankbuckley\TdsParserExceptionRepro\TdsParserExceptionRepro\Program.cs:line 127
   at Program.<Main>(String[] args)

[Wraith2]

It's hitting a very complicated edge case in the code changes I made for perf. Thankfully once you understand it the fix is relatively simple, just adding a little cleanup logic. I'll open a PR once I get it polished and tested.

@cheenamalhotra can you assign to me please

[frankbuckley]

It's hitting a very complicated edge case in the code changes I made for perf. Thankfully once you understand it the fix is relatively simple, just adding a little cleanup logic. I'll open a PR once I get it polished and tested.

Great! Thank you.

[frankbuckley]

@cheenamalhotra - when this merges is there a chance of a new preview release with this and the SqlVector tweaks?

[ErikEJ]

@frankbuckley This milestone for this is 6.1 (as you can see at the top of this issue)

And also 6.1 for this: #3468 (if that is what you are referring to?)

[Wraith2]

@cheenamalhotra - when this merges is there a chance of a new preview release with this and the SqlVector tweaks?

Depending on when they get merged you could take the build artifacts version from the later one which should contain all the fixes you need and try that.