TokenAuthIntegTests#testRefreshingMultipleTimesWithinWindowSucceeds failures

[jkakavas]

This has failed a few times already in 7.x and 7.7 https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(branch),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:testRefreshingMultipleTimesWithinWindowSucceeds),sort:!(time,desc))

Example failure : https://gradle-enterprise.elastic.co/s/m5xyt3as2qtv2/tests/g5u3xltc4r2ds-a52ia4rmdb2ji

This appears to not been failing in master ( after 450 iterations ) but reproducibly fails in 7.7 and 7.x once every 5 to 15 times. The difference between master and 7.x/7.7 is how we are checking that we can authenticate after the token is refreshed in that the test in master uses a rest client while the test in 7.7 uses a transport client. It's not clear at this time if this is a test issue in master that fails to test the scenario properly or a test bug in 7.x

I will mute this for now until we can get to the bottom of it

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[not-napoleon]

Found an example of it failing in master: https://gradle-enterprise.elastic.co/s/y3t5c3uc3fpni

I'm going to mute it in master as well.

[ywangd]

This is likely due to dirty read on the security index.

Assume node A holds the primary shard of the security index. Thread 1 hits node A and begins to index the new token document. While the indexing is still happening, thread 2 also hits node A and tries to get the document. Thread 2 may succeed and happily return since the document could be visible on node A's primary shard. But the indexing operation is not yet completed so it is not yet available on node B's replica shard. Now if thread 3 hits node B and tries to authenticate with the new token, it will fail because replica shard does not have the document yet. The described scenario is specific, but in general the failure can happen whenever two reads happen to hit shards that are not fully synced.

I can reproduce the failure reliably in 7.7 by changing numberOfThreads to 46 (higher number is more prone to trigger failure). And it can be fixed by re-use the same threadClient to perfom the authentication, i.e. threadClient.filterWithHeader(BEARER_TOKEN...).... The change is basically ensure authentication talks to the same server that answered the refresh call.

As for why it happens much more often with TransportClient compared to HLRC, I can only guess that TransportClient is faster and hence prone to this error. And thanks to @not-napoleon , now we know it can also happen to HLRC, just with a very low chance.

According to @jasontedor, there is no way to tell whether the read is stale or not. A possible fix might be swapping the order of "update exsiting token document" and "create new token document". This will guarantee that the new document is fully in place when "update to existing document" is visible.

PS: @gwbrown believes that this behaviour should be called "stale read" instead of "dirty read". I am using "dirty read" here since it is the term used in #52400.