Optimize performance of ES privilege response validation

[legrego]

Summary

Optimizes the performance of our response validation when we make a call to Elasticsearch's _has_privileges endpoint. Resolves #90016

This validation is currently performed by building a custom Joi schema. This works well most of the time, but does not scale when there is a large number of resources, due to the way we were building the validation schema.

My profiling indicates that it takes a lot of time to build the schema, but not a lot of time to actually perform the validation using the schema. As such, this PR changes the way the schema is built, but otherwise maintains parity with the existing implementation.

This PR moves the validation from Joi to @kbn/config-schema for a couple of reasons:

1. @kbn/config-schema is the preferred validation framework
2. It uses Joi under the covers, so the updated validation is very close the original implementation (low risk)
3. It allows for custom validation functions, which is something that we can technically do with Joi, but it is not as friendly to do.

How I tested

1. Start with a new ES Cluster (yarn es snapshot)
2. Start Kibana
3. Create 3500 spaces (I scripted using the spaces apis)
4. Attempt to load the space selector screen

---

Prior to this change, one such run showed that the entire request took almost 5 seconds, with 4.3 seconds spent in the response validation

[2021-02-02T18:49:59.441Z][DEBUG][plugins.spaces] SpacesClient.getAll(). querying all spaces
[2021-02-02T18:49:59.837Z][DEBUG][plugins.spaces] SpacesClient.getAll(). Found 3501 spaces.
query for spaces: 397.585ms
POST /_security/user/_has_privileges (22): 77.105ms
validateEsPrivilegeResponse (22): 4.351s
SpacesClient.getAll(): 4.889s

After this change, one such run showed that the entire request took under 1 second, with less than 500ms spend in the response validation

[2021-02-02T18:47:23.929Z][DEBUG][plugins.spaces] SpacesClient.getAll(). querying all spaces
[2021-02-02T18:47:24.037Z][DEBUG][plugins.spaces] SpacesClient.getAll(). Found 3501 spaces.
query for spaces: 110.866ms
POST /_security/user/_has_privileges (21): 78.987ms
validateEsPrivilegeResponse (21): 490.617ms
SpacesClient.getAll(): 828.321ms

[legrego]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jportner]

Looking good, I have one question below. Did not benchmark with Kibana running like you did, since I don't have that script handy.

[jportner]

LGTM!

[legrego]

Did not benchmark with Kibana running like you did, since I don't have that script handy.

I was going to package it up as part of this PR in case you wanted to test it yourself, but it's a messy little Node.js app that isn't suitable for anyone other than myself right now 🙂

[spalger]

jenkins test this

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: f16408e

Metrics [docs]

✅ unchanged

History

• 💚 Build #103591 succeeded 2db207d
• 💚 Build #103569 succeeded 2db207d
• 💚 Build #103278 succeeded e94bcff
• 💔 Build #103157 failed 0babfef
• 💔 Build #103137 failed bae9230

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream