Increase number of hostnames & rules in HTTPRoute

[achetronic]

What would you like to be added:

At this moment, number of hostnames and rules in HTTPRoutes are limited to 16 in both cases. I would like to have at least 100 of each one defined

Why this is needed:
This is a huge limitation for big companies with a big number of routes and prefixes.

[youngnick]

The reason we have the limits at the places they are at is the following:

• etcd has a maximum object size limit of 1Mbyte. After this, any updates to the object will fail to apply.
• adding more rules doesn't only mean adding more rules, it means adding more rules, x more matches, x more backends. So increasing these limits can have a massive impact on the overall size of the object, making it more likely that those limits will be hit.

Remember, the user experience here is that everything is working, you add that final extra rule, and then your updates fail until you get the size of the object below 1M. Note that this also includes the status, not just the spec.

We put the limits in place to encourage folks to shard their config across HTTPRoute objects. Because all HTTPRoutes are attached to Gateways anyway, the cost is some management overhead, balanced against the risk/certainty that people's HTTPRoutes will just start failing one day when some extra config is added.

Because of all of this, we're extremely reluctant to increase these limits.

[howardjohn]

Remember, the user experience here is that everything is working, you add that final extra rule, and then your updates fail until you get the size of the object below 1M

I'd argue the experience is the same today - you hit the arbitrary 16th rule and now updates fail. There are slightly fewer cases where it can be triggered but it's not that different.

If etcd limit is the real reason, shouldn't we look at the maximum possible size and object could be with the current limits, and if that is well below 1.5mb (usually it's 1.5mb not 1mb fwiw) we raise the limits?

I think more likely to be the limited is the CEL complexity limit which will kick in far sooner than etcd limits though obviously this is CRD development time vs at runtime. But the risk with CEL is that if we make it say 64 today and it's fine, it may prevent us from adding complex new fields in the backend. That being said, I suspect we are well below the complexity limits.

I would love to see a more data driven approach to our limits rather than one based on arbitrary numbers, by analyzing real sizing + CEL complexity budgets.

I do somewhat agree with the sharing argument -- a more critical limit, which I believe the original issue opener may be seeing if I'm reading between the lines, is the number of matches for a single rule which is harder to shard. And I think make the limit say 64 would still prevent crazy rules but make the 99% cases easier.

In my experience with our non-gateway APIs I have heard of 2 users with etcd size limits among 1000s, but almost every user working with gateway API ends up running into friction from the limits. I think meeting in the middle (say 4x pretty much every limit we have today) would go a long way towards removing friction while still allowing us to stay well within CEL and etcd limits