Wrong format of API key role descriptors used for node requests after cluster upgrade

[ywangd]

An effeciency optimisation for API Key role descriptors were introduced in #58156. The optimisation introdueced a new storage format for role descriptors in metadata of the authentication object. BWC was added in the PR. However it is not sufficient to cover the following scenario:

1. In v7.8, a task is created using API key which is serialised as part of the authentication header
2. The cluster is upgraded to v7.9
3. An upgraded node tries to run the task created in step 1, which in turn deserialises the authentication object. The task then needs to send requests using NodeClient, which serialises the authentication object again.
4. When the target node is also v7.9, the authentication header is sent without rewriting the API key role descriptors in the new storage format. However, the authetication object is declared to be v7.9.
5. When the target node recieves the request, it sees the authentication object is of format v7.9. Hence it tries to process the role descriptors in the new format way and fails because they are still in the old format of v7.8.

[elasticmachine]

Pinging @elastic/es-security (:Security/Authorization)

[albertzaharovits]

@ywangd
Can you please explain how this is possible in point 4 above?

However, the authetication object is declared to be v7.9.

[albertzaharovits]

I now get it, after I looked at the fix. It's a problem when we rewrite authentication to make it sure it is decodable by the remote node.