Blazor Server App fails to restart when restarted by IIS

[gregoryagu]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Steps to Reproduce:

1. Publish Blazor Server App running AspnetCore 6 to local IIS Server.

2. App runs successfully for some hours used by about 10 Customer Service reps on the local network.

3. IIS shuts down the app. Checking the Event Log shows an event is logged whereby the app crashed. Looking at the log of the crash shows this: IIS Worker Process stopped working and was closed. There is no more information as to why the Worker Process was stopped.

4. IIS restarts the app. The event log shows: Application '/LM/W3SVC/9/ROOT' started process '936' successfully and process '936' is listening on port '17093'.

5. The Blazor App throws an exception at StartUp. This is the exception:

Category: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware
EventId: 1
SpanId: 1c3119e0a4dcdf30
TraceId: 14a6bb3482129505bf214e80d29fed8a
ParentId: 0000000000000000
ConnectionId: 0HMG12RDCHRV2
RequestId: 0HMG12RDCHRV2:00000002
RequestPath: /

An unhandled exception has occurred while executing the request.

Exception: 
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.TryAdd(TKey key, TValue value)
   at Microsoft.AspNetCore.Components.Routing.QueryParameterValueSupplier.ForType(Type componentType)
   at Microsoft.AspNetCore.Components.RouteView.RenderPageWithParameters(RenderTreeBuilder builder)
   at Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder.AddContent(Int32 sequence, RenderFragment fragment)
   at CDS.Shared.MainLayout.BuildRenderTree(RenderTreeBuilder __builder)
   at Microsoft.AspNetCore.Components.ComponentBase.<.ctor>b__6_0(RenderTreeBuilder builder)
   at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment, Exception& renderFragmentException)
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.HandleException(Exception exception)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.HandleExceptionViaErrorBoundary(Exception error, ComponentState errorSourceOrNull)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.HandleException(Exception exception)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessPendingRender()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.AddToRenderQueue(Int32 componentId, RenderFragment renderFragment)
   at Microsoft.AspNetCore.Components.ComponentBase.StateHasChanged()
   at Microsoft.AspNetCore.Components.ComponentBase.CallOnParametersSetAsync()
   at Microsoft.AspNetCore.Components.ComponentBase.RunInitAndSetParametersAsync()
   at CDS.App.SetParametersAsync(ParameterView parameters)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.HandleException(Exception exception)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.HandleExceptionViaErrorBoundary(Exception error, ComponentState errorSourceOrNull)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.<WaitForQuiescence>g__ProcessAsynchronousWork|43_0()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.WaitForQuiescence()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderRootComponentAsync(Int32 componentId, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.CreateInitialRenderAsync(Type componentType, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.RenderComponentAsync(Type componentType, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<>c__11`1.<<InvokeAsync>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.ViewFeatures.StaticComponentRenderer.PrerenderComponentAsync(ParameterView parameters, HttpContext httpContext, Type componentType)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ComponentRenderer.PrerenderedServerComponentAsync(HttpContext context, ServerComponentInvocationSequence invocationId, Type type, ParameterView parametersCollection)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ComponentRenderer.RenderComponentAsync(ViewContext viewContext, Type componentType, RenderMode renderMode, Object parameters)
   at Microsoft.AspNetCore.Mvc.TagHelpers.ComponentTagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.<RunAsync>g__Awaited|0_0(Task task, TagHelperExecutionContext executionContext, Int32 i, Int32 count)
   at CDS.Pages.Pages__Host.ExecuteAsync()
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|28_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Note that there is no user code being run. It is unclear to me why the exception is thrown.
6. The exception is thrown repeatedly (100 times or more) until IIS Shuts down the site.
6A. I then manually restart the App using IIS.
7. The app then starts and runs successfully for several hours before repeating the above events.

Note that I have also reproduced this issue by manually recycling the AppPool. Also, when automatic recycling was turned on, it would sometimes. But this is not consistent. Sometimes it does start successfully when the AppPool is recycled.

8. This App also runs on a second IIS Server. Strangely, it has never crashed on this second server. I am not aware of any configuration differences on the second server. They both access the same database.

This is App.razor:

<CascadingValue Name="Authentication" Value=Authentication>
        <CascadingAuthenticationState>
            <Router AppAssembly="@typeof(App).Assembly">
                <Found Context="routeData">
                    @if (Authentication.IsAuthenticated)
                    {
                    <AuthorizeRouteView RouteData="@routeData" DefaultLayout="@typeof(MainLayout)">
                        <NotAuthorized>
                            <div class="container">
                                <div class="alert alert-info">
                                    <strong>Authorization Failure!</strong>@(Authentication.User!=null?Authentication.User.LoginName: "You") are not authorized to access this App. Please contact the Help Desk.
                                </div>
                            </div>
                        </NotAuthorized>
                    </AuthorizeRouteView>

                    }
                    else
                    {
                        <LayoutView>
                            <div class="container">
                                <div class="alert alert-info">
                                    <strong>Authentication Failure!</strong>@(Authentication.User!=null?Authentication.User.LoginName: "You") are not authorized to access this App. Please contact the Help Desk.
                                </div>
                            </div>
                        </LayoutView>

                    }
                </Found>
                <NotFound>
                    <LayoutView Layout="@typeof(MainLayout)">
                        <p>Sorry, there's nothing at this address.</p>
                    </LayoutView>
                </NotFound>
            </Router>
        </CascadingAuthenticationState>
    </CascadingValue>

This is App.razor.cs:

namespace CDS
{
    public partial class App
    {        
        [Inject]
        public CdsUserRepository CdsUserRepository { get; set; }   
        
        [Inject]
        AuthenticationStateProvider AuthenticationStateProvider { get; set; }

        [Inject]
        private IEventAggregator _eventAggregator { get; set; }

        private UserPrincipal _authentication;

        public UserPrincipal Authentication
        {
            get => _authentication ?? new UserPrincipal();
            set => _authentication = value;
        }          

        [Inject]
        public IDbContextFactory<CDSEntities> DbFactory { get; set; }

        public override async Task SetParametersAsync(ParameterView parameters)
        {
            await this.InitializeDataAsync();
            await base.SetParametersAsync(parameters);
        }
        public async Task InitializeDataAsync()
        {
            var authState = await this.AuthenticationStateProvider.GetAuthenticationStateAsync();
            if (authState is not null)
            {
                var userName = authState.User.Identity.Name;
                if (userName is not null)
                {
                    this.Authentication = new UserPrincipal();
                    this.Authentication.User = await this.CdsUserRepository.GetCdsUserByNameAsync(userName);
                    //Since the other components are not instansiated yet, this doesn't actually do anything useful.
                    if (this.Authentication.User is not null &&  this.Authentication.IsAuthenticated)
                    {
                        await this.GetUserRoles();
                        await this.GetFeatures();
                        var isLoggedInEvent = new IsLoggedInEvent();
                        isLoggedInEvent.UserPrincipal = Authentication.User;
                        await this._eventAggregator.PublishAsync(isLoggedInEvent);
                    }
                }
            }
        }
        private async Task GetUserRoles()
        {
            using var context = this.DbFactory.CreateDbContext();
            var query = context.CdsUserRole
                .Where(r => r.CdsUserId == Authentication.User.CdsUserId && r.ActiveFlag)                
                .Include(i=>i.CdsRole);

            var results = await query.ToListAsync();           

            foreach (var cdsUserRole in results)
            {
                this.Authentication.AppRoles.Add(cdsUserRole.CdsRole);
            }
        }
        private async Task GetFeatures()
        {
            using var context = this.DbFactory.CreateDbContext();
            var ids = new List<int>();
            
            foreach (var role in this.Authentication.AppRoles)
            {
                ids.Add(role.CdsRoleId);
            }
            var query = context.CdsFeatureRole.Where(i => ids.Contains(i.CdsRoleId)).Where(i => i.ActiveFlag).Include(i=>i.CdsFeature);
            var items = await query.ToListAsync();
            foreach (var cdsFeatureRole in items)
            {
                this.Authentication.Features.Add(cdsFeatureRole.CdsFeature);
            }
        }

    }
}

I include the app files to show what is going on there.

Any advice on how to resolve this would be appreciated.

Expected Behavior

After an IIS Restart, the app should start successfully.

Steps To Reproduce

See above

Exceptions (if any)

See above

.NET Version

6.0.1

Anything else?

This issue may have started after upgrading from .net 5 to 6. While running 5 for some month, this issue never occurred.

This is the full crash report:

CrashReport.wer.txt

[gregoryagu]

Thank you very much @javiercn