[FEATURE] When internal security configuration is updated, its should be confirmed to be a new value

[peternied]

Is your feature request related to a problem?

When looking into how SegRep [1] could impact the read consistency of the security configuration among the different nodes in a cluster, it exposes that on reload of configuration there is no mechanism in place to be sure the loaded configuration is actually newer.

What solution would you like?

When configuration updates the payload should include which configurations are to be reloaded, as well as an indicator of the change. In some systems this is an hash of the new config, ETAG, lastModified date or other value. We should include a value like this that is verified by nodes on reload of configuration that can be checked to confirm the configuration is at least as up to date as when the request was made.

Note; there should be wiggle room if multiple update configuration requests are in flight at once.

What alternatives have you considered?

The current system has no correctness assurances, we could continue doing nothing. Alternatively, from the node that processes the configuration update could query the other nodes in the cluster to be sure if they have gotten the update and then reissue updates until all nodes are complete.

Exit Criteria

• Author a test case that makes a configuration changes that will race against once another that will confirm configurations can get out of sync
  • An approach could be updating the same users password from each nodes at the same time, then check to see if each individual node has the same password value, or if any are different.
• Figure out how the best way an atomic change was made to the cluster
  • This problem isn't unique problem to security plugin's, maybe only triggering the fan out request from the cluster manager can ensure consistency
• Plumb through an identifier to validate the loaded config matched / doesn't match the expected
  • Possible prototype [FEATURE] When internal security configuration is updated, its should be confirmed to be a new value #3275 (comment)
• Add mechanism to resolved the mismatched configuration

Do you have any additional context?

• Concerns raised from segment replication changes [1] Compatibility with segment replication #2916

[peternied]

I've developed a prototype of this change that uses the sequence number from the configuration, might be viable to address this problem.

• main...peternied:security:verify-config-load

[stephen-crawford]

[Triage] Thanks for filing this issue @peternied. This issue has a clear solution outlined so seems actionable. Given the starting point Peter provided we can also mark good first issue and help wanted since that should be enough to point someone in the right direction.

[peternied]

I'm going to remove good first issue this is a complex distributed systems problem that needs to be piped through the security plugin - we wouldn't mind help, but I would not want someone to pick up this task without being familiar with the existing configuration system update process.

[peternied]

• Related issue [BUG] TransportConfigUpdateAction causing a race condition during node bootstrap leading to 403/500 errors #3204

[willyborankin]

@peternied I will take it after #4002 been merged