[API Key] Return base64 encoded API key when creating API key

[fkelbert]

Describe the feature: _security/api_key API should return the ready-to-use base64 encoded API key. This would many significantly more user-friendly. Currently, users need to manually create the base64 encoded API key from the id and and the api_key field.

Elasticsearch version (bin/elasticsearch --version): 7.5

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[albertzaharovits]

Thanks for bringing this to our attention @fkelbert !
We've discussed this before, and agreed this is desirable, but we didn't follow-up: #38291 (comment)

[aterreno]

That's a nice change, if anyone else was confused and came here googling from the documentation, you need to base64 the id and the api key appending the two separated by a ':'

like base64(id + ":" + apiKey)

[albertzaharovits]

Given our bwc goals, we have to maintain the existing fields in the response body. My proposal is to extend the response to add another field, named authorization, that contains the ApiKey authn scheme concatenated with the value of base64(id + ":" + api_key).
As an example, the current response for _security/api_key is :

{
  "id" : "Yw6PvXIBDLw0heIvcMR4",
  "name" : "my-api-key-3",
  "api_key" : "dvcZAa3LQe2RmyVcjLEpDw"
}

and I would propose we extend it to:

{
  "id" : "Yw6PvXIBDLw0heIvcMR4",
  "name" : "my-api-key-3",
  "api_key" : "dvcZAa3LQe2RmyVcjLEpDw",
  "authorization": "ApiKey WXc2UHZYSUJETHcwaGVJdmNNUjQ6ZHZjWkFhM0xRZTJSbXlWY2pMRXBEdw=="
}

The "authorization" field name is a bit funky. It is inspired on the HTTP request header name, which is how you'd utilize the API Key, eg curl -H "Authorization: ApiKey WXc2UHZYSUJETHcwaGVJdmNNUjQ6ZHZjWkFhM0xRZTJSbXlWY2pMRXBEdw==".

I would also say we remove the api_key field in the next major version (8).

@elastic/es-security what do you think about this proposal?

[ywangd]

I'd suggest something like:

{
  ...
  "credentials": "WXc2UHZYSUJETHcwaGVJdmNNUjQ6ZHZjWkFhM0xRZTJSbXlWY2pMRXBEdw=="
}

A few reasons:

• It's easy for user to add the ApiKey part. If user is parsing for this specific field, s/he is already aware this is for API keys.
• It is simpler to document this new field without the extra ApiKey prefix, which is already documented in the current documentation when explaining how to use the credentials into a curl call.
• The field name credentials is consistent with current documentation, specifically:

The API key returned by this API can then be used by sending a request with a Authorization header with a value having the prefix ApiKey followed by the credentials, where credentials is the base64 encoding of id and api_key joined by a colon.

I am more inclined to keep the api_key field:

• It seems to add unnecessary bwc burnden for users. It is like: we will add a new "convenient" field at the cost that your "inconvenient" way of working will break one day and have to update.
• The opposite of the current issue could happen, i.e. it is not friendly to have to decode the api_key from the base64 encoded value.

[jkakavas]

I only have a weak preference over this but I can't think of a valid use for the api key id or the api key value/secret on its own. I know that the get api key info or invalidate api key expects the id part but I don't remember why we did this and it doesn't make much sense now that I think about this. I expect callers of the get api key API to store either the base64 encoded concatenation string that they can then use, or store both the id and the secret in the same place eithe rway. What's more, it feels like the fact that we have an id and a value part is an implementation detail that we force to the users.

• It's trivial for us to parse the base64 encoded string and get the id part when api keys are used api key related APIs so we can handle the change easily
• Should we keep the api_key value and return the base64 encoded value there ? And deprecate and remove the id field , adjusting the other api key related APIs for 8 ?

[bytebilly]

I agree we should avoid the ApiKey prefix, since it makes it very related to the raw HTTP header value.
There are use cases (e.g. adding credentials to config files) that don't require the prefix, and users may be confused and copy&paste the entire string.

I also think that we should keep the api_key field. Users may use it in some scenarios (e.g. Elasticsearch language clients allow to specify separate values for id and api_key, and we may create an unnecessary change here. Also, users may think that we just renamed api_key, doing an incorrect double-encoding of base64(id:credentials).

We should find a good name for the new value, and even if being consistent with the docs makes some sense, I'd prefer to not consider it a requirement since we can change docs accordingly to the new name. The current mention in docs is not a JSON name, just a verbal term.

Said that, I'm not against using credentials. Other possible candidates may be token or auth, with the advantage of being shorter.

[jkakavas]

The more I think about this the stronger my preference gets I think..

I also think that we should keep the api_key field. Users may use it in some scenarios (e.g. Elasticsearch language clients allow to specify separate values for id and api_key, and we may create an unnecessary change here

My point is that these clients only do this because we forced them to do this by returning an id and a value in our API (and they opted in making it easier for the users as opposed to mandating that the users do the base64(id:key) ). I see this as a simplification that makes sense for 8 and not as an unnecessary change.

Other possible candidates may be token or auth, with the advantage of being shorter.

I would steer away from token as it can cause additional confusion ( relevant to our Token service and the tokens we create there ). Also, while I don't dislike the credentials or auth , the base64 encoded representation is the api key for all intents and purposes and I think we're trying to find ways around a decision we made in 6.x. Does anyone remember if there was a use case or a requirement that dictated the use/knowledge of 2 values on the client side ( apart from the implementation details on the server side ) ? Is the only argument in favor of maintaining and id and a value, to not make a breaking change in 8 ? ( Not saying this is not a valid argument on its own, I'm just not persuaded it is sufficient )

[bytebilly]

I see this as a simplification that makes sense for 8 and not as an unnecessary change.

I understand your point, but since we forced developers to deal with the initial choice, forcing them to change their code in 8.x is something that could create development effort (on their side).

I'm not sure the benefit (simplify the API) is higher than the cost (force users to change their code), even if I agree it would have been probably my first choice. I'm curious to get more context on the initial decision 🙂

If we'll go with removing the api_key field, I'd see just key as the right name for the new value. However in 7.x we would have api_key and key that is not that easy to decrypt from a user perspective.

[tvernum]

Does anyone remember if there was a use case or a requirement that dictated the use/knowledge of 2 values on the client side

Our APIs, by design, don't require that you know the credential in order to reference an API key.

We use Key Ids in:

1. The GET (list) api keys endpoint - we need something to return that uniquely identifies a key for which the secret is no longer retrievable
2. The DELETE api key endpoint - we need to be able to delete keys even if the owner does not has access to the secret
3. The SetSecurityUserProcessor - like (1), we need a unique id here
4. (Not yet) In audit logs, similarly to (3).

I feel strongly that we should not remove the identifier from create API Key response. It is the id that we will (& need to) use elsewhere and it should be explicit from the start.