@kbn/config-schema is slow

[kobelb]

At the moment, @kbn/config-schema is slow because Joi is admittedly slow, and not focused on increasing performance in the short-term: hapijs/joi#2340 (comment).

While performing some performance profiling for Fleet, I'm seeing around 6% of the total CPU time being spend performing validation with @kbn/config-schema.

I think we have two options here, embrace the fact that @kbn/config-schema is too slow for some purposes, and no longer use it situations like route validation. Change @kbn/config-schema to no longer rely on Joi, and work on improving its performance.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

I didn't investigate this properly, so I would have to verify it again to be sure, but I remember seeing a significant portion of startup CPU time is also spent inside @kbn/config-schema. It would be worth investigating this, if it's significant there would be even more motivation to increase the performance of @kbn/config-schema.

Having said that, I don't think it's worth for Kibana to maintain it's own validation library. Because config validation is part of the Core API we need to have control over that so it makes sense to use @kbn/config-schema, but instead of spending time making that faster and increasing the API surface, I think we should rather limit the time we spend on it and limit the API to only what's necessary for config validation.

[pgayvallet]

I think we have two options here, embrace the fact that @kbn/config-schema is too slow for some purposes, and no longer use it situations like route validation. Change @kbn/config-schema to no longer rely on Joi, and work on improving its performance.

Aren't both options the same? We will still need validation of some kind for routing, so both options means either finding a replacement for joi, or implement our own as config-schema underlying validation library, don't they? From what I've seen, some of our route validations are not 'less complex' that our config file validation, so we would need almost the same features for both.

Also, sorry to nitpick, but I'm not sure a 6% overheat issue can be labeled as bug? Isn't performance enough?

[kobelb]

Aren't both options the same? We will still need validation of some kind for routing, so both options means either finding a replacement for joi, or implement our own as config-schema underlying validation library, don't they? From what I've seen, some of our route validations are not 'less complex' that our config file validation, so we would need almost the same features for both.

In the context of route handling, both options are similar in that we don't want to use Joi to do route level validation. However, I still think there are two different options in regard to what we do with @kbn/config-schema as a whole.

1. Admit @kbn/config-schema is slow, and don't use it where performance matters
2. Fix @kbn/config-schema so it's now fast, can be used everywhere.

Also, sorry to nitpick, but I'm not sure a 6% overheat issue can be labeled as bug? Isn't performance enough?

I don't think we have a consistent way of labeling performance issues, so feel free to use whatever labels you desire :)

[pgayvallet]

Admit @kbn/config-schema is slow, and don't use it where performance matters

(not sure if I should comment here or in #78353, but) are all route validations concerned by this, or only specific bottlenecks?

If that's only bottlenecks, core's route validation system allows to use an arbitrary validation function instead of config-schema, so the quick win should probably be to use manual validation for these endpoints.

If all routes validations are considered as a potential performance issue, we need to find a faster alternative to joi , and document it as the preferred validation system for routing. But if we go down that road, 'Fixing' config-schema to change the underlying validation library is probably the way to go, as ideally, we wouldn't be using two distinct libraries.

As a sidenote, if we keep config-schema for route validation, we might want to rename it to something like @kbn/validation-schema or something.

[mshustov]

We will still need validation of some kind for routing

I think @rudolf meant that we shouldn't enforce @kbn/config-schema as a standard for the routing validation. However, we already allow plugins to use any custom validation library. My main concern about the topic: what if a plugin is not a hot path and called not so often as Fleet endpoints? They might be interested in using whatever tool off-the-shelf and don't spend their time investigating alternative options.

From what I've seen, some of our route validations are not 'less complex' that our config file validation, so we would need almost the same features for both.

Another concern is the tooling fragmentation. It's almost impossible to make sure all the validation library implements the same security model (for example, filtering out the raw values from error messages #58843).

While performing some performance profiling for Fleet, I'm seeing around 6% of the total CPU time being spend performing validation with @kbn/config-schema.

Is it a bottleneck for Fleet? We might need to add a benchmark setup to get absolute numbers to understand which order of numbers we are talking about - millions or thousands of operations per second. Probably, updating the Joi version helps us to increase that limit to reasonable values.

[kobelb]

(not sure if I should comment here or in #78353, but) are all route validations concerned by this, or only specific bottlenecks?

This issue is meant to address all usages of @kbn/config-schema everywhere. Built-in route validation, custom route validation, config, etc.

If that's only bottlenecks, core's route validation system allows to use an arbitrary validation function instead of config-schema, so the quick win should probably be to use manual validation for these endpoints.

Agreed, this is possible for the custom validation for the routes. This is what we did for Fleet, no longer use @kbn/config-schema or Joi for the route validation.

I think @rudolf meant that we shouldn't enforce @kbn/config-schema as a standard for the routing validation. However, we already allow plugins to use any custom validation library.

If we decide that @kbn/config-schema is slow and we aren't going to invest the effort to make it fast, I think we should stop encouraging plugins to use @kbn/config-schema for their route validation. With the way the API is currently created, it leads plugin developers to default to using @kbn/config-schema

My main concern about the topic: what if a plugin is not a hot path and called not so often as Fleet endpoints? They might be interested in using whatever tool off-the-shelf and don't spend their time investigating alternative options.

If a usage of @kbn/config-schema isn't in the hot path, it's not as large of a concern. However, I don't think we should be encouraging developers to use something that is slow and won't scale.

Is it a bottleneck for Fleet? We might need to add a benchmark setup to get absolute numbers to understand which order of numbers we are talking about - millions or thousands of operations per second. Probably, updating the Joi version helps us to increase that limit to reasonable values.

Yes, it is a bottleneck. The current bottleneck is because of the use of @kbn/config-schema for the built-in route validation, which is why I created #78353 to work-around this specific bottleneck.

[mshustov]

Yes, it is a bottleneck. The current bottleneck is because of the use of @kbn/config-schema for the built-in route validation, which is why I created #78353 to work-around this specific bottleneck.

@kobelb I'm going to create a benchmark (after FF) to compare @kbn/config-schema with joi, io-ts to have particular numbers we can discuss here.

[afharo]

Sharing an existing benchmark I found comparing joi, io-ts and others: https://github.com/gcanti/io-ts-benchmarks

[mshustov]

3 years ago

ok, might require some work to actualize 😅