Developer UX: Reducing the need for infrastructure awareness for Developers

[shaneutt]

In the post-1.2 world we've been thinking a lot about Gateway API User Experience (UX) and how we can improve that specifically for developers.

As a developer with Ingress you would package your Deployment which let people know how it was run on Kubernetes, and then packaged your Ingress to describe how it does traffic. This way does not require developers to think much about the infrastructure (e.g. the Gateway). The parentRef relationship with Gateways in HTTPRoute is more complicated.

The question here is: whether we could significantly improve user experience if we had some mechanism in place such that the developers who publish *Routes with their applications did not have to bother with parentRefs.

How that ends up looking doesn't concern me nearly as much as whether other people feel similarly. Please let us know: does this resonate with you? Do you think having the "simple ingress experience" more clearly paralleled in Gateway API would significantly increase adoption?

[keithmattix]

Personally, I don't think the adoption friction is the Gateway API itself but rather the market forces of:

1. Kubernetes having a single blessed reference implementation
2. the API coming installed by default.

#3384 covers 2) so I'll focus on 1). If I were a betting man, I'd wager that the vast majority of the Ingress API (today and historically) comes from ingress-nginx and not by a narrow margin. In fact, right before Gateway API took off, the ecosystem was coalescing around controller-specific CRDs because of Ingress's limitations. Therefore, my conclusion is that Ingress's adoption has nothing to do with UX but rather the absence of needing to choose an implementation. To be blunt: I suspect that Gateway API's adoption will always be somewhat capped because ingress-nginx (not the Ingress API) provides a Kubernetes org-blessed option that no one will get fired for picking.

Apologies for the tangent, but TLDR - if the end goal is to increase Gateway API adoption, then this discussion is asking the wrong question IMO

[shaneutt]

I think the community is divided on this @aojea. I would characterize my own take on this as "we should aim to absorb as many Ingress v1 users as are in need of service beyond what Ingress provides, but full absorption is probably not feasible in any kind of near term and should not be an explicit goal". It may be something worth us taking some time to discuss this point further and distill that into an even more definitive and official stance on it in our documentation.

[aojea]

move folks from ingress to gateway api, especially since ingress doesn't seem to be adding new features.

exactly my question, how many of those users not need new features and just current Ingress works ...

[youngnick]

For many people, current Ingress "works" with a bunch of annotations. The amount of use cases that base Ingress is useful for is not that big.

However, I think the GA API guarantees make it clear that base Ingress will be around approximately forever, but if you want more complicated things that require annotations on Ingress, you should really be looking to move to Gateway API. The annotations are the key.

[keithmattix]

Should we be encouraging to gateway implementations about supporting Ingress then? Because if users do indeed need just a simple api without any of bells and whistles, there shouldn't be much holding us back from a healthier ingress implementation ecosystem

[arkodg]

thanks for raising this point @keithmattix.
Speaking on behalf of an implementation, Envoy Gateway, which only supports the Gateway API - we've asked users with existing Ingress resources to convert their config (manually or using ingress2gateway) if they wanted to use Envoy Gateway.
This has probably led to some drop off, but our assumption was, with Kubernetes / SIG-Network continuing to promote Gateway API as the L7 API to adopt (for greenfield as well as brownfield), this drop off would be temporary, and those users would eventually end up migrating to Gateway API.
If the stance from SIG-Network is closer to - existing users with simple use cases, its ok to use Ingress and for for the rest, use Gateway API, then implementations such as Envoy Gateway will need to consider also supporting the Ingress API, which will in turn affect the entire ecosystem where Gateway API adoption will slow down and may only be relevant for greenfield cases

[mikemorris]

Do you think having the "simple ingress experience" more clearly paralleled in Gateway API would significantly increase adoption?

I don't think this would have a direct impact on most end-user adoption by proprietary app developers or platform teams (for whom the persona model can be an improvement over a tightly-controlled monolithic ingress - I have not heard distributed app-specific bundled Ingress as a common pattern for these larger organizations).

I do think this could be beneficial for open source project owners responsible for bundling and publishing their projects with supplemental resources such as Helm charts who would prefer to minimize additional friction in an onboarding/quick-start setup.

[mlavacca]

Since one of the main Gateway API differentiators from Ingress is the isolation of concerns between the involved personas, the concept of making the app developer completely unaware of the underlying infrastructure resonates a lot with me. This is what we state in the description of Ana, the app developer, in our documentation:

... She is in a unique position among Gateway API personas, since her focus is on the business needs her application is meant to serve, not Kubernetes or Gateway API. In fact, Ana is likely to view Gateway API and Kubernetes as pure friction getting in her way to get things done.

Since from Ana's point of view Gateway API is pure friction, I do totally agree that we should do as much as we can to reduce that friction as much as possible, and removing the NEED to set Gateway references from the routes, looks a good step forward to me.

[youngnick]

Yeah, okay, that makes sense and would work okay - because if there's no Gateway, then there's no relationship, and the Route won't be accepted by any implementation.

[mlavacca]

Yes, exactly. The greater impact given by this proposal is that it'll be no longer possible to have HTTPRoute without ParentRefs. I was pretty sure that ParentRefs is a mandatory field, but it turned out to be an optional one:

gateway-api/apis/v1/shared_types.go

Lines 220 to 226 in 8285041

	// +optional
	// +kubebuilder:validation:MaxItems=32
	// <gateway:standard:validation:XValidation:message="sectionName must be specified when parentRefs includes 2 or more references to the same parent",rule="self.all(p1, self.all(p2, p1.group == p2.group && p1.kind == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && p1.__namespace__ == p2.__namespace__ )) ? ((!has(p1.sectionName) || p1.sectionName == '') == (!has(p2.sectionName) || p2.sectionName == '')) : true))">
	// <gateway:standard:validation:XValidation:message="sectionName must be unique when parentRefs includes 2 or more references to the same parent",rule="self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName == '')) || (has(p1.sectionName) && has(p2.sectionName) && p1.sectionName == p2.sectionName))))">
	// <gateway:experimental:validation:XValidation:message="sectionName or port must be specified when parentRefs includes 2 or more references to the same parent",rule="self.all(p1, self.all(p2, p1.group == p2.group && p1.kind == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) || p1.sectionName == '') == (!has(p2.sectionName) || p2.sectionName == '') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) || p2.port == 0)): true))">
	// <gateway:experimental:validation:XValidation:message="sectionName or port must be unique when parentRefs includes 2 or more references to the same parent",rule="self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port == p2.port))))">
	ParentRefs []ParentReference `json:"parentRefs,omitempty"`

This means that as for now it's possible to create an HTTPRoute without Parents at all. With this change, such a scenario will no longer be possible. I personally don't see it as a problem, but I may be missing some context here. Is there any specific use case related to an HTTPRoute with no parents at all?

[youngnick]

Oh, interesting, I didn't realize we had that set as optional. In the end, I guess it doesn't matter, because if there's no chain of ownership, then it won't be picked up for processing anyway.

Note that this does mean that setting a default will need to be done at the parent layer (set a default for the []ParentReference field), not inside the ParentReference struct, or else it won't work like we want.

[keithmattix]

So essentially what's being proposed is a semantic for a default gateway that exists in the same namespace as the route. That feels fine if, as @mlavacca points out above, we make parentRef mandatory (not sure why it isn't already?). My only concern is whether we'll be having this same conversation in 3 years because users want to have the default gateway live in a different namespace than the route (or some other feature) and we find ourselves trying to make things "easier" again.

[mlavacca]

I think the goal of this discussion is to understand whether we want to make life easier for potential new users who can get hit by the complexity of Gateway API. My point of view is that no step in that direction has to be precluded a priori. I'd love to discuss all the possible improvements of the API without prejudice, not only feature-wise but UX-wise as well. In case we get to the point when we discuss making users' lives easier when referencing Gateways in different namespaces, I'll be happy to have that discussion, and in case we'll be able to find a solution that makes sense and does not undermine the API design and integrity, well, that's a win to me :)

Of course, API principles integrity is paramount, but let's not shy away from exploring potential improvements and evaluating them individually.

[kflynn]

It's nice to see this discussion happening and getting engagement. Overall I think @mlavacca is on to something promising: his proposal seems to do a nice job of using the existing machinery and preserving existing semantics, while also giving Ana a helpful default that lets her focus on her business needs rather than forcing her to think about infrastructure.

That said, @youngnick has done his usual (depressingly 🙂) good job of pointing out pitfalls. True to form, I shall try to tease these apart by talking about, you guessed it, use cases! 😂

1. Ana might create a Route that stomps her colleagues Amy's Route.

   This is... let's call it a "concern by design": if you're allowing self-service route configuration, this is always going to be a thing that can happen, whether or not parentRef is optional. Ana can stomp Amy's Route right now, in fact, she just has to supply the right parentRef (which is likely to be trivial).

   Our primary protection against this right now is Kubernetes, in the form of RBAC and namespaces (if you want to make sure that Ana can't stomp on Amy's Routes, give Amy her own namespace and don't let Ana create Routes there!). That'll continue to work under Mattia's proposal, and it'll still work if it's some evildoer (let's call them Erin) creating the stomp Route rather than Ana. Gateway API needs to trust that the cluster under it is functional, and that's OK.

2. Suppose our evildoer, Erin, tries to stomp all over everything by creating a new Gateway that exposes all of Ana's Routes on some other port, with auth turned off.

   This is a much more interesting concern. It's not hard for Erin to create an evil-erin namespace and install a Gateway in it with allowedRoutes set to allow Routes from Ana's namespace. It wouldn't have any effect right now because the parentRefs of Ana's Routes won't point to Erin's Gateway, but if we just declare that parentRef is optional and a missing parentRef means "any Gateway", then this attack would work beautifully.

   Mattia's proposal addresses this by sidestepping that "any Gateway" behavior, instead giving a missing parentRef a much narrower scope that's limited to the same namespace as the Route. Of course, if Erin can create a Gateway in Ana's namespace, this attack will still work, but we're back to being able to defend against it with RBAC and namespaces.

3. On the other hand... suppose a customer comes up to Ian and Chihiro to say that they desperately need access to the cluster's application from their shiny new VPN, they need it yesterday, and their compliance auditors demand that they must use the NewShinyVPN Gateway for access -- it's OK if the existing Gateway is still there for everyone else to use, but they must use a NewShinyVPN Gateway or nothing.

   In this case, the best of all worlds would be if Ian and Chihiro could create a NewShinyVPN Gateway instance that would snap up all the Routes -- otherwise, they'll have to edit every single Route in the cluster. This is a place where Nick's question about having a way to configure the default for a missing parentRef really shines: if we had a way to properly do that, this use case becomes quick and easy and Ana needn't even think about it.

   And since this is admittedly a stretch goal in my mind (though an interesting one to think about), a more realistic place where Nick's suggestion about configuring the default is if Ian and Chihiro would really prefer to have one Gateway for the cluster that all namespaces default to, rather than needing one Gateway per namespace.

4. Finally, uh... what about GAMMA?

   Probably the most sane way to handle GAMMA, at least in the first pass, is to continue requiring a parentRef for it. That'll work fine under Mattia's proposal, and it'll still allow Ana to save time for north-south Routes.

   In the longer term, we need to circle back to the question of how to get GAMMA and north-south to truly work seamlessly together, and I look forward to kicking this around some in Salt Lake with @mikemorris and anyone else who's interested. 🙂

[youngnick]

Perhaps it makes sense by defining how much friction is acceptable for Ana. And then having a high bar to change that definition. Things can always be made easier, but the cost of doing so is not always obvious up front

Yeah, I tend to agree. I think it's inevitable that, for a very complex API like this, some additional complexity management is required on Ana's part. I think that setting a default value for the parentRef if not specified is probably a reasonable approximation here, and doesn't preclude us also doing things to make the whole flow better in consumer config like Helm charts.

[kflynn]

There are a couple of really interesting things in here that, again, I'd kind of like to separate.

@youngnick writes:

But the advantage we get out of that is that there is a clear audit trail for every HTTPRoute in the cluster as where it's associated - right in the HTTPRoute.

This is actually fascinating to me, because I view parentRef as a statement of intent rather than part of the audit trail. The audit trail is the status stanzas. We hammer on this all the time when talking to people about Gateway API: if things feel weird, first thing you do is you check the status! and we do that because it's easy to get the parentRef wrong, but the status reflects truth.

The status will still reflect the truth if we define a default for parentRefs, and I think that there's a worthwhile distinction to be drawn between defining default semantics if you specify no parentRefs and magic. Magic would be picking a Gateway based on, I dunno, what we believe your workload does. That would clearly be madness. 🙂 No one is suggesting that. I rather liked that @mlavacca's proposal was far from that.

To @keithmattix's point about languages, C definitely had a refreshing lack of spooky action at a distance. 🙂 And it had a slew of segfaults, buffer overruns, etc., right? Moving into a world where the developer didn't have to explicitly always type out everything to do with memory absolutely improved the world, even though you can likely still find C veterans who think that Go's garbage collector is too much magic for them to be fully comfortable with.

Ultimately, the evolution of languages tends to show that requiring developers to take on the burden of explicitly stating everything pushes them away from using the thing that requires that. I'd like us not to do that here. I also think it's worth noting that defining default semantics is easier on developers in practice than defining default values that show up when you read the resource back: having default values appear on readback makes a lot of CD tools very unhappy (so, for example, right now, AFAIK any developer using ArgoCD must include the matches stanza in their HTTPRoutes because otherwise ArgoCD will think that the resource is always out of sync 🙁).

[shaneutt]

requiring developers to take on the burden of explicitly stating everything pushes them away from using the thing that requires that.

This resonates very strongly with me, and I think fits very well with the original spirit of the post. I wouldn't say that we've quite managed to build strong consensus on the principle yet, and this comment to me is a helpful highlight on what that principle is.

[keithmattix]

I'm somewhat persuaded by that idea @kflynn and @shaneutt. I do feel like there's a balancing corollary to that principle though that I can't quite put my finger on, but it goes something like allowing developers to not specify enough (for some value of enough) will push away all other personas down the line. There's a harmony between those two principles that we need to aim for and I think @mlavacca's proposal does seem to be a good step in that direction

[kflynn]

@keithmattix I might phrase that concern as "ease of use is important, but magic isn't supportable". You're absolutely right about the balancing act required to get it right -- computing really isn't a field where we often have the luxury of unambiguously correct answers. 🙂

[kflynn]

Also, to @youngnick's point about conflict resolution -- the conflict resolution semantics are a huge help, definitely! In practice, you can still do some pretty nasty stompage by applying a route that's more specific, rather than identical.

The updated Gateway API workshop for Salt Lake actually shows this: we have a canary GRPCRoute with no matches clause, just a parentRef to a particular Service. Works great. We then apply a second GRPCRoute with the same parentRef, but this one specifies a matches clause to pick a specific gRPC method. That's not seen as a conflict, even though traffic to that gRPC method dutifully follows the new route, not the old route.

To be clear, I think of this as a feature: it makes a lot of things much simpler than they'd otherwise be! It's probably also an area where having something to show the routing table could be a great addition.