Add spec.api.onlyBindToAddress configuration

[pschichtel]

I picked up #1038 and rebased it onto main. Smoke tests pass, but not entirely clear what was left here.

From #1038:

Should a new bindAddress option be created and change documentation for the existing address option, or should the existing address option's behavior be changed to match the existing documentation?

The later I can see causing some issues with existing setups, bindAddress is probably the safest route. Currently the default for bindAddress is "0.0.0.0", which is the default kube-api-server uses if the --bind-address flag isn't used.

Issue Fixes #957

What this PR Includes Adds the ability to define a bindAddress for the kube-api-server.

[twz123]

Thanks for picking this up.

I'd double-check if there are some bad interactions with node-local load balancing? I imagine the LB config needs to pick up the correct bind addresses as well?

We should think about some integration test, too, I suppose.

[twz123]

Okay, some things to proceed with:

• We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.
• We need to double-check if this plays nicely with nllb and other things. Do we need to update the docs on e.g. external load balancing? How would this affect k0sctl, if at all?

[pschichtel]

We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.

I'm not very familiar with the code base yet, can you point me to examples for integration tests?

We need to double-check if this plays nicely with nllb and other things.

I'd have to look into how nllb works exactly. Depending on how the upstream connection is made its configuration probably needs adjustments.

Do we need to update the docs on e.g. external load balancing?

I guess a disclaimer like "if you change the bind address make sure to align your external load balancer" or so. I feel like people changing bind addresses are probably aware of the impact. But that's just my opinion.

How would this affect k0sctl, if at all?

k0sctl would need to interact with the API server on the correct IP, wouldn't it?

[twz123]

Sorry for getting back to you so late @pschichtel. I'll bluntly blame the holiday season for that. 😅

We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.

I'm not very familiar with the code base yet, can you point me to examples for integration tests?

We need to double-check if this plays nicely with nllb and other things.

I'd have to look into how nllb works exactly. Depending on how the upstream connection is made its configuration probably needs adjustments.

We could try to start with the existing nllb inttest, copy that over into a separate directory and start hacking on the new one. You can probably delete all the cluster-restart stuff in there, and just rely on the cluster bootstrapping part. You need to add the new directory name to inttest/Makefile.variables. Then you should be able to do something like make check-yolo (or whatever the name is you choose.)

Might make sense to simply try to overwrite the bind addresses of every component that you can get hold of and see what happens to the cluster.

Do we need to update the docs on e.g. external load balancing?

I guess a disclaimer like "if you change the bind address make sure to align your external load balancer" or so. I feel like people changing bind addresses are probably aware of the impact. But that's just my opinion.

Sounds good enough for the start.

How would this affect k0sctl, if at all?

k0sctl would need to interact with the API server on the correct IP, wouldn't it?

Yes. Would it need to look at the k0s config to figure this out? Or would it "just work"? Maybe @kke can tell.

[pschichtel]

Sorry for getting back to you so late @pschichtel. I'll bluntly blame the holiday season for that. 😅

No worries, I don't have any rush with this one.

We could try to start with the existing nllb inttest, copy that over into a separate directory and start hacking on the new one. You can probably delete all the cluster-restart stuff in there, and just rely on the cluster bootstrapping part. You need to add the new directory name to inttest/Makefile.variables. Then you should be able to do something like make check-yolo (or whatever the name is you choose.)

Might make sense to simply try to overwrite the bind addresses of every component that you can get hold of and see what happens to the cluster.

Makes sense, I'll have a go at that.

[twz123]

(The linter error will go away after a rebase. See #3991 for details.)

[pschichtel]

yeah I plan to work on this again towards the end of this week

[andycandy-de]

Any idea when this feature will be available? I also have multiple IPs and only want to bind the k0scontroller to a specific IP.

[pschichtel]

@andycandy-de sadly this had to be pushed back a little in my prio list, but I still intent to eventuelly finish this up. I think some of the recent k0s/k8s releases also added additional components, which probably needs/deserves some investigation. I'm currently unable to work on it until mid of April, so no sooner than that.

[pschichtel]

it's been months of rebasing now.

[twz123]

Thanks for your dedication and perseverance, and sorry for the very late feedback. Looks good. Just some small fixes to use net.JoinHostPort. I'd like to get some quick feedback from other maintainers on the name of the config knob, 👍 otherwise.

[twz123]

Would be great to add some details as to why this knob exists, and what may be the consequences if set to true. Basically some excerpt of #3824 (comment).

[pschichtel]

I improved it I think, please have a look.

[twz123]

As it defaults to false and it's omitempty, the k0s kubeconfig create command won't include that field.

Suggested change

onlyBindToAddress: false

[twz123]

I wonder if we could just keep bindAddress instead of onlyBindToAddress and let the user decide which address to bind to instead of forcing it to be either the advertised address or all interfaces. This may be a niche requirement, though. I cannot really judge. Maybe some folks with more network expertise have an opinion? OTOH, the address field will be defaulted by k0s's address discovery if omitted. There wouldn't be an easy way to let users use the auto-detected address as the bind address, then.

Anyway, I'm fine with the onlyBindToAddress logic as well, but wanted to at least ask about it.

[jnummelin]

IMO bindAddress makes more sense and is probably more common in general. At least I remember seeing such knobs on other things quite often. 😄

[twz123]

Agreed. However, what about

There wouldn't be an easy way to let users use the auto-detected address as the bind address, then.

[jnummelin]

hmm, what's wrong in the current default behaviour of autodetecting the publish address and binding to all interfaces?

[pschichtel]

IMO bindAddress makes more sense and is probably more common in general. At least I remember seeing such knobs on other things quite often.

bind-address is a common term yes, but rarely any software asks you to correctly configure several addresses for different purposes, that may or may not interact with each other. That's why I intended to reduce the bind-address parameter to what makes sense. I think the current documentation of address is misleading, given nothing actually binds to that address, it's just the internal address published to other components.

hmm, what's wrong in the current default behaviour of autodetecting the publish address and binding to all interfaces?

The original issue #957 describe where this PR comes from: Multi-homed kubernetes hosts. In my case specially, my k0s hosts was also my edge router, so one of the interfaces the API server bound to was my internet uplink. With proper firewall rules in place no big deal, but it carries some risk. My setup since changed (for the better).

There wouldn't be an easy way to let users use the auto-detected address as the bind address, then.

if the user configured bindToAddressOnly: true without actually configuring the address, then it would just use the auto-detected address, right? (probably worth documenting too)

As I said in #3824 (comment), address and bindAddress are strongly related. Configuring them incorrectly will break the cluster. You can't bind to an address that's not internally announced (unless NAT between components is supported, which would be ridiculous), so you have 2 options: bind to all interfaces and announce one of the addresses or bind to the address you announce. I can't see any other valid configuration (again: assuming NAT is not a thing here).

Also... I'd assume (I don't have numbers on it) fairly few people actually have a need for this. Few kubernetes hosts are multi-homed and of those most are fine with binding all interfaces.

[twz123]

An alternative name could be bindToAllInterfaces which defaults to true. This would have to be implemented as a bool pointer though. onlyBindToAddress is a bit harder for me to understand.

[pschichtel]

I read in other PRs that you generally prefer defaulting to false, so I'd assume other booleans are also like that. Not sure this is worth breaking the consistency, especially since this is more of a niche option.

[kke]

bindExclusivelyToAddress, bindExclusive, bind(To)Address(Only), addressBind?

Somehow onlyBindToAddress sounds clunky to me.

Of course address (advertise) + bindAddress: <address> (bind) would be the best, I think there's an issue for that (#1880).

[twz123]

Of course address (advertise) + bindAddress: <address> (bind) would be the best, I think there's an issue for that (#1880).

Still unsure if we care about the autodetection aspects. We'd loose those with address + bindAddress. Personally, I'm okay with the boolean flag (whatever name it will have in the end). We can revisit that whenever we get around to #4822.

[pschichtel]

Somehow onlyBindToAddress sounds clunky to me.

I agree, I'm not super happy with the name, but at least it's descriptive.

Of course address (advertise) + bindAddress:

(bind) would be the best

I actually don't agree here. I think bindAddress is not an option that should be offered, because the additional flexibility brings no benefit, only the risk that people misconfigure it.

[pschichtel]

Great. I'm currently on the go, but can probably clean this up on Sunday.

[github-actions]

This pull request has merge conflicts that need to be resolved.

[github-actions]

This pull request has merge conflicts that need to be resolved.

[pschichtel]

Is that failing job just flakyness? It seems to exceeded a deadline unrelated to my change, right?

[twz123]

Yep, the inttests are quite a bit flaky, unfortunately.

[twz123]

Thank you!