ASP.NET Core - Performance - Load Testing issue Microsoft.AspNetCore.Server.Kestrel[13] Connection id "xyz", Request id "abc" An unhandled exception was thrown by the application.

[sanasz91mdev]

Hello,

testing an ASP.NET Core API under load gives following error:

**fail:** _Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLTTTNCMN23D", Request id "0HLTTTNCMN23D:00000002": An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content._

Load test settings: 4000 concurrent requests coming each second for a duration of 1 minute

I am using JMeter for load testing. This issue does not arise for 2000 concurrent users.

[analogrelay]

We don't see this in our load testing (with many more concurrent requests). Can you provide a runnable application and steps for reproducing this?

[sanasz91mdev]

@anurse it was a simple API calling a stored procedure and returning response.
Do you need code for this?

[davidfowl]

@sanasz91mdev we can't take action because we can't reproduce it. Can you provide the application and the error you saw?

[sanasz91mdev]

@davidfowl .. provide your email so that i can add you to access to my code?

Check question for error:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HLTTTNCMN23D", Request id "0HLTTTNCMN23D:00000002": An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.

[Roman-Blinkov]

I saw this exception in logs. In my case no load was applied.

@sanasz91mdev Lets try to replace stored procedure with await Task.Delay(100) and do public simple hello-world instead of complex app.

[xlievo]

@davidfowl can you have a look? thank you :)
same question!
simple hello-world app

os: FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine

Load test settings: 1000 concurrent users for 1 minute

This exception will occur for external network access, and there will be no pressure problem for local access :)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLU8P7S8VQMH", Request id "0HLU8P7S8VQMH:00000001": An unhandled exception was thrown by the application.
System.InvalidOperationException: The examined position cannot be less than the previously examined position.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_InvalidExaminedPosition()
   at System.IO.Pipelines.Pipe.AdvanceReader(BufferSegment consumedSegment, Int32 consumedIndex, BufferSegment examinedSegment, Int32 examinedIndex)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.<ReadAsyncInternal>g__ResetReadingState|9_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at BusinessController.Call() in /root/workspace/log/Common/Common.cs:line 613
   at lambda_method(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[xlievo]

The only answer I can think of is: Internet bandwidth is full, and local access does not need bandwidth
Will this exception be thrown when the server bandwidth is full?
But I'm not sure！

[davidfowl]

Sorry that doesn’t help. You need to provide at a bare minimum a repro of what the application is doing and how you’re running the load test

[sanasz91mdev]

application is just calling a stored procedure and Load testing is being done by JMeter over same network @davidfowl

[rclarkezaner]

I am getting this too.

Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Ubuntu 18.04.2
Apache/2.4.29

I had wondered if this error was related to the size of the request.

It is not. The crash here the content-length is 1694976

I have had successful requests more than 3x that size

Here is some data captured by my UseExceptionHandler(), in case it helps someone

The examined position cannot be less than the previously examined position.

   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_InvalidExaminedPosition()
   at System.IO.Pipelines.Pipe.AdvanceReader(BufferSegment consumedSegment, Int32 consumedIndex, BufferSegment examinedSegment, Int32 examinedIndex)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.<ReadAsyncInternal>g__ResetReadingState|9_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.TryCreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageBinderFactory.<>c__DisplayClass2_0.<<CreatePropertyBinder>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.BindArgumentsCoreAsync()
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)`

Method: POST
Url: http://app.example.com/
getRequest: The examined position cannot be less than the previously examined position. getting Form
Request.Headers:
Cache-Control: max-age=0
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Cookie: _ga=GA1.2.105213241.1586789971; _gid=GA1.2.1505485800.1586789971; .AspNetCore.Antiforgery.o4DuEIf4Zck=CfDJ8GTEBDREdqBAl7oEJcwj9RDOYrAXKfGtBrSVximKJXggJziZDCe2AAyvoPeSdKCkUZAkO4Mm7LYluYEOECmWqxYartqysCxjE0wfl_yHCYZBLE03eTm2PHH0OEae0QSVhgIa0G3uB8LvEuNqfJZRcJk
Host: example.com
Referer: https://app.example.com/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Upgrade-Insecure-Requests: 1
Origin: https://app.example.com
Content-Length: 1694976
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: example.com
X-Forwarded-Server: example.com

[davidfowl]

@rclarkezaner Can you reproduce this after a single request? Can you reproduce it reliably? Does it happen with form posts only? (I see the error is caused by a post request here).

[rclarkezaner]

@davidfowl Unfortunately I cannot reproduce it. I only see it happen on form posts. It's an online application form where customers enter biodata and upload a selfie. Sometimes it will happen several times in a row to the same customer making me think it's caused by something in the request body.