Migrating Kibana system features from 6.8.20 fails with reindexing error

[sabarasaba]

Found this error while testing system indices migration:

1. Set up a 6.8.20 cluster and run Kibana to create Kibana indices
2. Upgrade to a 7.16 cluster
3. Optionally, run Kibana 7.16
4. Trigger a system feature migration, either from the upgrade assistant or by invoking the POST API directly

See the following failure using the GET system feature migration status API:

JSON response

{
    "features": [
        {
            "feature_name": "kibana",
            "minimum_index_version": "6.8.20",
            "migration_status": "ERROR",
            "indices": [
                {
                    "index": ".apm-agent-configuration",
                    "version": "7.16.0"
                },
                {
                    "index": ".apm-custom-link",
                    "version": "7.16.0"
                },
                {
                    "index": ".kibana_1",
                    "version": "6.8.20",
                    "failure_cause": {
                        "error": {
                            "root_cause": [
                                {
                                    "type": "exception",
                                    "reason": "error occurred while reindexing, bulk failures [{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"space:default\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"maps-telemetry:maps-telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"config:6.8.20\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"telemetry:telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403}], search failures []",
                                    "stack_trace": "ElasticsearchException[error occurred while reindexing, bulk failures [{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"space:default\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"maps-telemetry:maps-telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"config:6.8.20\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"telemetry:telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403}], search failures []]\n\tat org.elasticsearch.upgrades.SystemIndexMigrator.logAndThrowExceptionForFailures(SystemIndexMigrator.java:532)\n\tat org.elasticsearch.upgrades.SystemIndexMigrator.lambda$migrateSingleIndex$12(SystemIndexMigrator.java:413)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.lambda$finishHim$3(AbstractAsyncBulkByScrollAction.java:568)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource.cleanup(ClientScrollableHitSource.java:129)\n\tat org.elasticsearch.index.reindex.ScrollableHitSource.lambda$close$2(ScrollableHitSource.java:115)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:116)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:112)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.action.search.ClearScrollController.onFreedContext(ClearScrollController.java:133)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$2(ClearScrollController.java:117)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1549)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1529)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41)\n\tat org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$3(SearchTransportService.java:454)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:341)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:389)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:386)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:136)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:128)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:415)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:1037)\n\tat org.elasticsearch.transport.TransportService.access$100(TransportService.java:66)\n\tat org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:143)\n\tat org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167)\n\tat org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874)\n\tat org.elasticsearch.action.search.SearchTransportService.sendFreeContext(SearchTransportService.java:130)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$4(ClearScrollController.java:114)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.search.SearchScrollAsyncAction.collectNodesAndRun(SearchScrollAsyncAction.java:116)\n\tat org.elasticsearch.action.search.ClearScrollController.cleanScrollIds(ClearScrollController.java:106)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$new$1(ClearScrollController.java:72)\n\tat org.elasticsearch.action.search.ClearScrollController.run(ClearScrollController.java:81)\n\tat org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:39)\n\tat org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:19)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.support.AbstractClient.clearScroll(AbstractClient.java:637)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource.clearScroll(ClientScrollableHitSource.java:112)\n\tat org.elasticsearch.index.reindex.ScrollableHitSource.close(ScrollableHitSource.java:115)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.finishHim(AbstractAsyncBulkByScrollAction.java:560)\n\tat org.elasticsearch.reindex.Reindexer$AsyncIndexBySearchAction.finishHim(Reindexer.java:244)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.refreshAndFinish(AbstractAsyncBulkByScrollAction.java:522)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.onBulkResponse(AbstractAsyncBulkByScrollAction.java:472)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:421)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:417)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.finishHim(Retry.java:168)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:112)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:71)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620)\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:818)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.runReroutePhase(TransportReplicationAction.java:227)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:222)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:85)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:590)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:736)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:279)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:192)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:200)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57)\n\tat org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:492)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.execute(Retry.java:201)\n\tat org.elasticsearch.action.bulk.Retry.withBackoff(Retry.java:51)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.sendBulkRequest(AbstractAsyncBulkByScrollAction.java:417)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.prepareBulkRequest(AbstractAsyncBulkByScrollAction.java:396)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$1.doRun(AbstractAsyncBulkByScrollAction.java:345)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:771)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.RunOnce.run(RunOnce.java:29)\n\tat org.elasticsearch.index.reindex.WorkerBulkByScrollTaskState$DelayedPrepareBulkRequest.lambda$new$0(WorkerBulkByScrollTaskState.java:254)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:712)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
                                }
                            ],
                            "type": "exception",
                            "reason": "error occurred while reindexing, bulk failures [{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"space:default\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"maps-telemetry:maps-telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"config:6.8.20\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"telemetry:telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403}], search failures []",
                            "stack_trace": "ElasticsearchException[error occurred while reindexing, bulk failures [{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"space:default\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"maps-telemetry:maps-telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"config:6.8.20\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_1-reindexed-for-8\",\"type\":\"_doc\",\"id\":\"telemetry:telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_1-reindexed-for-8] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403}], search failures []]\n\tat org.elasticsearch.upgrades.SystemIndexMigrator.logAndThrowExceptionForFailures(SystemIndexMigrator.java:532)\n\tat org.elasticsearch.upgrades.SystemIndexMigrator.lambda$migrateSingleIndex$12(SystemIndexMigrator.java:413)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.lambda$finishHim$3(AbstractAsyncBulkByScrollAction.java:568)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource.cleanup(ClientScrollableHitSource.java:129)\n\tat org.elasticsearch.index.reindex.ScrollableHitSource.lambda$close$2(ScrollableHitSource.java:115)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:116)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:112)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.action.search.ClearScrollController.onFreedContext(ClearScrollController.java:133)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$2(ClearScrollController.java:117)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1549)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1529)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41)\n\tat org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$3(SearchTransportService.java:454)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:341)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:389)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:386)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:136)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:128)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:415)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:1037)\n\tat org.elasticsearch.transport.TransportService.access$100(TransportService.java:66)\n\tat org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:143)\n\tat org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167)\n\tat org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874)\n\tat org.elasticsearch.action.search.SearchTransportService.sendFreeContext(SearchTransportService.java:130)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$4(ClearScrollController.java:114)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.search.SearchScrollAsyncAction.collectNodesAndRun(SearchScrollAsyncAction.java:116)\n\tat org.elasticsearch.action.search.ClearScrollController.cleanScrollIds(ClearScrollController.java:106)\n\tat org.elasticsearch.action.search.ClearScrollController.lambda$new$1(ClearScrollController.java:72)\n\tat org.elasticsearch.action.search.ClearScrollController.run(ClearScrollController.java:81)\n\tat org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:39)\n\tat org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:19)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.support.AbstractClient.clearScroll(AbstractClient.java:637)\n\tat org.elasticsearch.index.reindex.ClientScrollableHitSource.clearScroll(ClientScrollableHitSource.java:112)\n\tat org.elasticsearch.index.reindex.ScrollableHitSource.close(ScrollableHitSource.java:115)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.finishHim(AbstractAsyncBulkByScrollAction.java:560)\n\tat org.elasticsearch.reindex.Reindexer$AsyncIndexBySearchAction.finishHim(Reindexer.java:244)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.refreshAndFinish(AbstractAsyncBulkByScrollAction.java:522)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.onBulkResponse(AbstractAsyncBulkByScrollAction.java:472)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:421)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:417)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.finishHim(Retry.java:168)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:112)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:71)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620)\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:818)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.runReroutePhase(TransportReplicationAction.java:227)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:222)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:85)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:590)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:736)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:279)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:192)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:200)\n\tat org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57)\n\tat org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)\n\tat org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:492)\n\tat org.elasticsearch.action.bulk.Retry$RetryHandler.execute(Retry.java:201)\n\tat org.elasticsearch.action.bulk.Retry.withBackoff(Retry.java:51)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.sendBulkRequest(AbstractAsyncBulkByScrollAction.java:417)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.prepareBulkRequest(AbstractAsyncBulkByScrollAction.java:396)\n\tat org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$1.doRun(AbstractAsyncBulkByScrollAction.java:345)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:771)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.RunOnce.run(RunOnce.java:29)\n\tat org.elasticsearch.index.reindex.WorkerBulkByScrollTaskState$DelayedPrepareBulkRequest.lambda$new$0(WorkerBulkByScrollTaskState.java:254)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:712)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
                        }
                    }
                },
                {
                    "index": ".kibana_1-reindexed-for-8",
                    "version": "7.16.0"
                },
                {
                    "index": ".kibana_7.16.0_001",
                    "version": "7.16.0"
                },
                {
                    "index": ".kibana_security_session_1",
                    "version": "7.16.0"
                },
                {
                    "index": ".kibana_task_manager_7.16.0_001",
                    "version": "7.16.0"
                },
                {
                    "index": ".kibana_task_manager_pre7.4.0_001",
                    "version": "7.16.0"
                }
            ]
        },
        {
            "feature_name": "security",
            "minimum_index_version": "6.8.20",
            "migration_status": "MIGRATION_NEEDED",
            "indices": [
                {
                    "index": ".security-6",
                    "version": "6.8.20"
                }
            ]
        }
    ],
    "migration_status": "ERROR"
}

If it's of any help, here's the snapshot I created in 6.8.20 that you can drop in 7.16 to test it out 6.8-data-snapshot.zip

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

@sabarasaba I wasn't able to replicate this while doing very simple things with my own 6.8 setup, so I'll see if I can figure out what is going on with your snapshot.

[williamrandolph]

It looks like running Kibana 7.16 is necessary for reproducing this issue. Kibana's seems to run migrations that leave old indices in the cluster with write blocks on them. I'll keep working on this.

[cjcenizal]

@sabarasaba I also noticed that the error reporting isn't very helpful. I spun up a 6.8.latest deployment, installed all the sample data, upgrades to 7.16.0, and here's the error I see in UA when I run the system index migration:

Can we do anything to surface more helpful information?

[sabarasaba]

@cjcenizal Good point! I've created elastic/kibana#119429 to keep the discussion there.

@williamrandolph Will the changes you made to fix this issue also address the bug that CJ encountered? 🤔