Long lived api keys/tokens

[jaymode]

The security codebase already supports the use of access tokens for
interactive access. These tokens are based on the OAuth2 spec; there is
a short-lived access token and a refresh token with a lifetime of 24 hours.
These tokens are not suitable for Hadoop jobs or recurring scheduled tasks.

Hadoop jobs typically require a token that may need to be refreshed but
the token needs to remain the same after refresh; our current tokens
always generate a new access and refresh token upon refresh. Another
pattern of Hadoop jobs is that once they are complete the tokens are
not invalidated but instead rely on a expiration time of something like
a week.

The security tokens are not a fit for recurring/scheduled jobs as they
can only be refreshed for 24 hours. If the system that would refresh the
token is down for greater than 24 hours, which could be possible over a
weekend or holiday, the jobs would require updating to function again.

After discussion, we've settled upon adding long lived tokens that would
be akin to API tokens. The properties of these tokens are as follows:

• Tokens will store the authentication information of the user that
  created them
• By default, tokens have no expiration. A cluster level setting will
  be available to specify a maximum lifetime
• When creating a token, a maximum lifetime can be specified for cases
  where the token is expected to stop working after a certain amount of
  time

In terms of APIs, we will be adding:

Create a token with optional expiration, which is a TimeValue.

POST /_security/token
{
  "expiration": "7d"
}

List security tokens. Optionally restrict by username and/or realm.

GET /_security/token
{
  "username": "john.doe",
  "realm": "ldap1"
}

List specific token

GET /_security/token/<token>

Remove a token.

DELETE /_security/token/<token>

In order to use these tokens, they will be passed over HTTP using the
Authorization header. The value must be prefixed with Bearer .

In order to use this token, security must take the value and find the
document with this id. If the document does not exist or the search fails,
the token may not be used. Once security has retrieved a document with
this ID, it must make sure to verify the token has not expired.

Invalidation or a removal of a token should remove the token document
from the security index.

Tokens must store enough information to re-build authentication and
support determining if the token has expired.

Future considerations:

• The ability to limit the scope of a token
• keep track of last used time

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

@jbaiera and @chrisdavies FYI. Let me know if you see any issues with this proposal.

[jasontedor]

@martijnvg @ywelsch Let us watch this issue in the context of CCR where we would consider using these tokens instead of the current approach of associating authorization headers with the persistent task.

[jbaiera]

• By default, tokens have no expiration. A cluster level setting will
  be available to specify a maximum lifetime
• When creating a token, a maximum lifetime can be specified for cases
  where the token is expected to stop working after a certain amount of
  time

Are we thinking that the cluster setting for maximum token lifetime will be the absolute maximum even if a client requests a longer lifetime? So if the maximum TTL for a token is set at 7 days in the cluster settings, and a client requests a token that lives for 1 month, will the client receive a token that expires in 7 days or 1 month?

Not sure if this was already assumed, but when we receive a token, it should probably denote its expiration date and time. I didn't see any example responses, so I figured I would put that here.

Final thought - It might make sense to allow clients to name the tokens to make interacting with and managing responses from the token api easier.

[jaymode]

Are we thinking that the cluster setting for maximum token lifetime will be the absolute maximum even if a client requests a longer lifetime?

Yes.

So if the maximum TTL for a token is set at 7 days in the cluster settings, and a client requests a token that lives for 1 month, will the client receive a token that expires in 7 days or 1 month?

I'd prefer for this to be an error condition, but given the variability in what a user could define the client would receive a token that expires in 7 days.

Not sure if this was already assumed, but when we receive a token, it should probably denote its expiration date and time

Makes sense.

Although the more I think about this, I am contemplating leaving out this ability in the first iteration and see what kind of feedback we get.

Final thought - It might make sense to allow clients to name the tokens to make interacting with and managing responses from the token api easier.

++

[jaymode]

The ability to define scoped access is allowed by providing a role definition with the token but is not necessarily a role that can retrieved using another mechanism. In order to support this, we need to think about how this role is handled in a few different scenarios:

1. Same node
2. Node to node in the same cluster
3. Node to remote node

The current thinking for 1 is that the role would be loaded by the service that loads the token. We need a way to get this role into the authorization service. The ThreadContext is the quick way out but I'd like to avoid using this as it feels like a mistake. We may be able to make use of metadata on the AuthenticationResult object that was recently added.

For the node to node scenario, there is a question of whether the receiving node should perform the loading of the role or if it should be serialized with the request and token. We've never serialized a role like this before, so we would be adding to the data that one nodes trusts from the other.

In terms of cross cluster, we've always just sent the role names and relied on the remote cluster looking this up. I think we set a dangerous precedent if we decide to serialize the role. However, if we do not serialize the role then these tokens cannot be used for cross cluster jobs, which hinders their use significantly.

[tvernum]

Invalidation or a removal of a token should remove the token document from the security index.

It's an implementation detail, but I think we should keep the tokens in an "invalidated" state for a short period (~24h?).
My two reasons for that are:

• Forensics. If invalidating a token removes all record of it, then it might be used by someone to cover their tracks. 24h is long enough for an admin to generate audit logs of whatever they think they need to retain.
• Simple debugging. It's nicer to be able to diagnose (in the server logs) that a job attempted to use a token after it was "deleted" as opposed to using a token that never existed.