Limit number of concurrent authentication requests when the cache expires

[tvernum]

Many of our realms extend CachingUsernamePasswordRealm which maintains a local (per node) cache of users and credentials so that we do not need to do a lookup on the underlying datastore for every request.

This is particularly important when the datastore is external, e.g. an LDAP directory. If the cache was not there we would need to do an LDAP bind (and potentially also group/metadata lookups) for every REST request.

This cache has a per-entry expiry, where an entry is a "user". That means that when the cache entry for a user expires, all authentication requests for that user will lookup the original datasource until the cache is populated again (when the first authentication succeeds).

Additionally, various other factors can cause a cache entry to be evicted (e.g. we invalidate on incorrect password so that we can check for an updated password in the datastore, and there is an API to clear the cache by realm/user).

Consequently, for a single user on a single node, there is a period where we do not make requests to the datastore, and then a short period where we make potentially multiple requests.

e.g. If somone has a number of beats running on various source systems, and those beats connect directly to Elasticsearch using a single user defined in LDAP, then every 30 minutes or so, every one of those beats could trigger an LDAP bind causing cyclic spikes in load on the LDAP server.

This behaviour is problematic, but the solution is not obvious.

• We can limit the impact of this through connection pooling (and have done so), but this doesn't reduce the total number of requests to the LDAP directory, it just flattens & stretches the spikes as that there are less simultaneous requests, but no change in the total number of requests.

• We could implement a "cache loader" approach, where only 1 task can be attempting to load a cache entry for a given key. That is, if we find that the cache does not have an entry for a specific user, then we perform a lookup on the original datasource, and any other lookups for that same user will wait for the first lookup to complete.
  This sort of concurrency behaviour is difficult to do well and can easily be a source of subtle bugs, particularly as we need the "wait for cache to populate" to be asynchronous.

• An alternative that I have a prototype for, is to simply limit the number of concurrent authentication requests that a single realm can perform simulatenously (with a BlockingQueue to hold pending requests). This limit is global and does not care about the identity of the authenticating user. It sits before the cache, so the first N requests will execute and populate the cache as needed, and then subsequent requests will be released from the queue and find the cache is ready and the relevant user entries are populated.
  This works quite well when a realm is authenticating a small number of distinct users, but those users perform multiple simultaneous requests to Elasticsearch. When there are multiple users involved, it can also help, but the result is very similar to using a connection pool.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

I haven't looked at the code yet, but I have qualms moving the blocking queue before the cache. If we are facing the surge, instead of the downstream LDAP server, we are inherently exposing ourselves to DOS . If the blocking queue is before the cache the DOS would impact all the users in the realm, if for example there's a misconfigured beats somewhere. Considering how realms are chained, a DOSed LDAP realm could escalate into a total mess, were users are authenticated by whatever realm is not busy.

I think we should take the hard route of limiting the number of outgoing resolve requests per user.

[bizybot]

This cache has a per-entry expiry, where an entry is a "user". That means that when the cache entry for a user expires, all authentication requests for that user will lookup the original datasource until the cache is populated again (when the first authentication succeeds).

I hope I understood the problem, following is an alternative solution:
Do something like Guava Cache expireAfterAccess - expire entries after the specified duration has passed since the entry was last accessed by a read or a write. Though this might mean that in case of beats there is a possibility of the information remains cached for long duration as periodically accessed and not refreshed from source. But this could be mitigated by refreshAfterWrite which queries ldap and updates cache entry async. To avoid surge the value chosen for duration for refresh after write could be random between a range so avoiding big sudden spikes to source.

I have not looked at the Cache APIs so not sure if these are possible with ES cache implementation but we could add that functionality if not present.

[jaymode]

DOS is a concern, but I think there are many other ways to DOS a cluster so I wouldn't consider that a reason to not consider a proposed solution.

I feel like there are two items to consider, number of concurrent authentications overall and number of concurrent authentications per user. The number of overall concurrent authentications can (and I think should) be handled at a higher level in the authentication service, with a thread limited executor (and careful handling of async actions using the same executor with higher priority).

The per user concurrent authentications should be handled at the realm level. As @tvernum mentioned, this is complicated by the asynchronous nature. I have some ideas that I can explore for this.

[s1monw]

FYI I removed the discuss label seems legit.

[tvernum]

The number of overall concurrent authentications can (and I think should) be handled at a higher level in the authentication service

I think there are valid use cases for either approach of limiting it globally, or per realm.
I don't think we should implement both but I would like us to seriously consider which one we do.

Doing it globally allows us to segregate authentication (including CPU expensive tasks like password hashing) from the other uses of the generic thread pool (& potentially freeing up some CPU time for other tasks)

Doing it per realm allows the "administrative" realms (e.g. reserved/file) realm to be isolated from the "user" realms (LDAP/AD/etc).

I don't know that one is better than the other, but it's worth being clear about our reasoning for picking one over the other.