improve flux manager with custom flags

[ghost]

What would you like Renovate to be able to do?

Good day,

I would like to be able to add flags for the flux manager. Currently, flux is always running the following command.

renovate/lib/manager/flux/artifacts.ts

Line 19 in a98e7af

const cmd = `flux install --export > ${quote(packageFileName)}`;

It would be really nice if you could add some flags. So you can update your flux components even more personalized.

Something like flux install ${costume_flags} --export > ${quote(packageFileName)}

If you have any ideas on how this should be implemented, please tell us here.

My idea is to add the option to set flags. The flags will be used if the manager runs the update command.

"flux": {
    "fileMatch": [
        "(^|/)flux-system/gotk-components\\.yaml$"
    ],
    "flags": [
        "--network-policy=false"
    ]
},

Like this, it's possible to update flux and ignore the network policy.

Is this a feature you are interested in implementing yourself?

No

[rarkins]

I think we'd want it to be more descriptively named, such as binaryOptions, and perhaps an array of strings, where the meaning of its value can differ between managers (and so would need to be documented by each). Flux could be the first manager, but this will need an external contribution.

[onedr0p]

@gopherpro for now I'm using $patch: delete on the kustomizations so the network policies are not created. For example:

https://github.com/onedr0p/home-ops/blob/main/cluster/base/flux-system/gotk-patches.yaml

and

https://github.com/onedr0p/home-ops/blob/main/cluster/base/flux-system/kustomization.yaml

[kingdonb]

An influx of users came to know Flux and Renovate work together; we discovered this issue report this morning, when a user mentioned that Renovate PR was removing their image-automation-controller and image-reflector-controller.

I don't think something as broad as a full-blown escape hatch where users can put whatever options they want is necessarily the best solution here, there is some header information in comments of manifests of newer Flux versions and they tell what components have been installed. So it could be figured out by a parser and the options added dynamically to the update PR without user input.

The default Flux bootstrap runs with this option at its default value:

--components=[four main components of flux]

(this flag can almost always be omitted, unless users wanted to run with fewer than the 4 default controllers – you can determine this from the headers) and users may add two additional components to support image automation with:

--components-extra=image-automation-controller,image-reflector-controller

Here is the header in the manifest as an example:

https://github.com/kingdonb/csh-flux/pull/30/files#diff-b068612a3a8295bc24e2998c37a5dd7378da25ea603bbc17eea659c777365432L4

I'm not sure how long ago this header was added, but almost every Flux installation will have these comments. If they are missing, I think it would run with the defaults.

[onedr0p]

@kingdonb nice to see you here and would love to see Renovate mentioned in the Flux docs!

While parsing the file is fine to determine what components are installed and maybe acting on it, the comments are not great at determining any other flux install flags, e.g. --network-policy=false

[kingdonb]

@onedr0p I personally use the $patch: delete method for that as you mentioned. Users can delete or patch any part of Flux with a kustomization.yaml and patchesStrategicMerge section, or other patch, but they cannot request a part to be added from upstream if it has been omitted altogether.

This is the only critical change needed before I think we can start advertising Renovate to Flux users. (In fact we added Renovate already to the docs this morning, but backed it out until this issue can be addressed.)

• Back out renovatebot until upstream issue 13952 is fixed fluxcd/website#744

[onedr0p]

I do not mind using the $patch: delete however it does make me more cautious on Flux updates in case new network policies are added, or their names are changed. It doesn't seem like you can write "delete all network policies from a file" with kustomize, unless I missed that in the docs somewhere ;)

But with that said I usually read the Flux release notes and check the diffs, so it's not super important to me, but maybe others.

Ideally having Renovate check the comments and adding those additional components doesn't seem too hard to implement.

[kingdonb]

@onedr0p I am not fully expert in Kustomize (if not me, then who is? seriously though...) but I have seen wildcards used in some surprising places.

I just checked, and $patch: delete is not one of them that supports it, so you're right. That's a bit risky.

I think this issue report asked for the feature and I think it got lost in the translation because it looks like the related issue has been merged and this still is not a feature. Maybe there's another method I'm not aware of where you can already do this.

• Allow setting name to "*" for bulk editing kubernetes-sigs/kustomize#567

[onedr0p]

Yep, there doesn't seem to be a way to wildcard or use regex on that. :(

[danports]

Interesting...I didn't know that Components comment existed, probably because I didn't specify --components or --components-extra when I installed Flux on my clusters. I agree that parsing that comment in order to provide more accurate arguments to flux install would be ideal. Beyond this, flux install supports several additional parameters like --cluster-domain and --registry - it's quite possible we could extract them all from the existing manifest with the appropriate regexes, which may obviate the need for something like binaryOptions altogether.

[kingdonb]

I always point people at this section of our docs, and try to highlight the "Automated Upgrades" section, which points out how to use Flux and GitHub Actions together to keep Flux up to date. This always feels like a forgotten corner of our docs, and when I do this, I specify the --components-extra field manually, writing it by hand into the GitHub Action yaml.

I'd rather point at a full-featured solution that doesn't stand awkwardly by itself as a one-off and has other uses, as Renovate clearly does. This seems like a great tool to integrate with Flux, and I've even used it in some of the maintenance of Flux v1 before, I already recognize the value of this tool. I don't think I understood that it's fully open source before.

This is really great, if that's so!

https://fluxcd.io/docs/installation/#bootstrap-upgrade

☝️ Here is the doc from Flux docs, where we currently describe "Automated Upgrades" in a manner that's loosely related to this idea: https://fluxcd.io/docs/use-cases/gh-actions-auto-pr/ – basically just what Renovate does, only much more limited (it opens pull requests.)

I'll have to explore this in more depth later on this week or next. I'd love to help validate that Flux and Renovate can be used together as much as possible without friction!

[rarkins]

Hi, Renovate maintainer here. Not a Flux user yet but happy to advise on elegant ways to support this capability once the requirements are nailed down. My preference when possible is that the "source of truth" for such things is in the tool's own config, package or lock files so that it's not necessary to duplicate it in Renovate config and potentially drift later on.

[kingdonb]

There has been a discussion here or there about having a Flux config object which contains global settings for the whole of Flux, so that Flux can be configured on platforms which do not use bootstrap (there's been some discussion about how to configure Flux on platforms like OpenShift where you get to expose your service as a "Subscription" but it doesn't accept configuration params, or other installation methods like the Flux terraform provider) can all be configured in a same way, but so far this concept hasn't got any traction with the maintainers of Flux.

The officially supported way to install Flux is the bootstrap method via the CLI, and the official way to configure it is by passing options in, with an official method of patching described in the docs that explains how to configure Flux things that are not supported by bootstrap flags.

These are the Flux official configuration methods to know. The relevant flags to cover, if you're interested in covering all flags, are on flux install:

Flags:
      --cluster-domain string
      --components strings
      --components-extra strings
      --export
  -h, --help
      --image-pull-secret string
      --log-level logLevel
      --network-policy
      --registry string
      --toleration-keys strings
  -v, --version string
      --watch-all-namespaces

network-policy and watch-all-namespaces are the only "binaryOptions". I would not consider it wrong to support these, but if your preference is to infer all configuration from files in the repo, the only one that I can easily extract that information in a parseable way are --components and --components-extra.

There are 6 Flux components, two of them extra, and that has been a stable list since Flux v2 was created. It seems reasonable to accept the default list is likely to remain stable forever, and any new components will likely be considered extra ones, so you can simply parse this list and filter out the default controllers, then include any extras that you found in components-extra. (I would validate that they are of the regex shape [a-z-,]+ or perhaps more generally a valid deployment name and don't pass anything which didn't conform with this, to avoid creating a potential attack vector.)

It would be nice to support removing controllers from components but pretty low rating IMHO on the "nice to have" scale – not likely to be a heavily used feature, those 4 are core components of Flux and most users will need them all.

I don't know how you get away without tightly coupling to the CLI if you want to support Flux broadly with all custom flags from now on, I probably wouldn't entertain that if I was a Renovate maintainer. The --components-extra is the only flag that I would advise supporting first before all other flags, and it has a source of truth in the file gotk-components.yaml:

https://github.com/kingdonb/bootstrap-repo/blob/9975294044fb568e3f5700e6085a833937acadc0/clusters/moo-cluster/flux-system/gotk-components.yaml#L1-L4

[kingdonb]

I am still learning typescript, so I will try to get around to making this change (if nobody beats me to it first!)

[danports]

@kingdonb Be my guest! My recent contributions to Renovate were my first time writing TypeScript, and I found it enjoyable.

@rarkins Agreed, much better to keep the configuration DRY.

Yes, I'd advocate for an iterative/flag-by-flag approach to adding Renovate support for the various flux install options, starting with --components-extra.

It would be useful if flux install/flux bootstrap wrote the flags to the manifest header...then we could kill all of the birds with one stone. (This has the advantage of being simple and not requiring the addition of a Flux global settings system which hasn't been a popular idea anyway.) However, that wouldn't address all of the various manifests already out there in the wild. Perhaps we could add support to Renovate for extracting the most common flux install options, and then update Flux to write the flags to the file header to capture everything going forward.

[danports]

To elaborate on that last idea (though this should probably be in an issue on the Flux repo), we could have Flux write a header like this, which would serve both our purposes here and also (I think) provide value to Flux users who are not (yet 😉) using Renovate:

# This manifest was generated by flux. DO NOT EDIT.
# Flux Version: v0.26.2
# Components: source-controller,kustomize-controller,helm-controller,notification-controller,image-reflector-controller,image-automation-controller
# To upgrade your Flux installation in the future, update the flux CLI and then run:
# flux bootstrap github --owner=my-github-username --repository=my-repository --components-extra=image-reflector-controller,image-automation-controller <...other flags here...>

[kingdonb]

The only reason I think that wouldn't be done is because the flux bootstrap flags are not necessarily easily translated into flux install flags.

In principle it's a fine idea, but what we're really after are the flags to pass to flux install --export so that it generates a gotk-components.yaml which is in the same vein as our existing one. The upgrade wouldn't change gotk-sync.yaml and it wouldn't run bootstrap again, the upgrade is for the components only, not the syncs.

But that's a nit! Let's start with the low-hanging fruit, I like the plan as it's been laid out 👍

[viceice]

It will be really great if there is an option to configure the flux install custom flags

Please don't post me too comments. Use 👍 to vote for the issue. Otherwise you'll make a lot noise for subscribed users.

[danports]

@kingdonb Any updates? I don't want to rush you, but it also seems like this is a significant issue for Flux users on Renovate, so if you haven't gotten to it, I'd like to at least knock out the initial --components-extra fix myself soon.

[kingdonb]

I have been back to this thread a couple times but haven't made any progress on it myself. Have at it!

[danports]

OK, no problem. #14332 addresses --components. The other options to consider support for appear to be --cluster-domain, --image-pull-secret, --network-policy, --registry, --toleration-keys, and --watch-all-namespaces.

[Diaoul]

I would love support for --cluster-domain, how hard would it be to implement? Can you give some pointers? 🙏

[viceice]

I would love support for --cluster-domain, how hard would it be to implement? Can you give some pointers? 🙏

• feat(flux): Generate system manifests using the components specified in the manifest header #14332

[Diaoul]

Thanks, it's slightly different though as we cannot extract the cluster domain value from a comment in the header.

• it's possible to find it as part of an URL in the gotk-components.yaml file but it's a bit wonky
• is there a way to pass options to the manager to "hardcode" the cluster domain and use that when generating the new one?

Let me know if any of this would be acceptable.

[kingdonb]

It would be great to have an allow-list of options that you can pass to flux install --export from the list of options supported by Flux. The people who pass extra options to bootstrap or install, would be the best to know which options are necessary.

Is there any example of a parameterized Renovate manager that I can take a look at to understand what's possible?

Flags:
      --cluster-domain string      internal cluster domain (default "cluster.local")
# (already supported)
###      --components strings         list of components, accepts comma-separated values (default [source-controller,kustomize-controller,helm-controller,notification-controller])
##      --components-extra strings   list of components in addition to those supplied or defaulted, accepts values such as 'image-reflector-controller,image-automation-controller'
##      --export                     write the install manifests to stdout and exit
##  -h, --help                       help for install
      --image-pull-secret string   Kubernetes secret name used for pulling the toolkit images from a private registry
      --log-level logLevel         log level, available options are: (debug, info, error) (default info)
      --network-policy             deny ingress access to the toolkit controllers from other namespaces using network policies (default true)
      --registry string            container registry where the toolkit images are published (default "ghcr.io/fluxcd")
      --toleration-keys strings    list of toleration keys used to schedule the components pods onto nodes with matching taints
##  -v, --version string             toolkit version, when specified the manifests are downloaded from https://github.com/fluxcd/flux2/releases
      --watch-all-namespaces       watch for custom resources in all namespaces, if set to false it will only watch the namespace where the toolkit is installed (default true)

I've marked off all of the options that Renovate wouldn't need to expose, the parameters that could be safely left out from the list exposed by the manager without hampering any Flux user capabilities.

There's a further list of "global flags" also accepted by flux install, but I don't think any of them are applicable as they would be used for talking to the cluster, and flux install --export is just talking to stdout.

[onedr0p]

I don't think this is needed anymore as you can "take over" Flux with Flux and not rely on the gotk-components.yaml file anymore. I would like to see these system manifests deprecated in Renovate and focused more on OCIRepository, HelmRepository and GitRepository resources.

Install Flux

kubectl apply --server-side --kustomize path/to/bootstrap/folder/below

# bootstrap.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/fluxcd/flux2/manifests/install?ref=v0.38.2
# patches:
# patch out wha you want to change

Then you can take Flux over with either option below which are supported by Flux now:

1. OCIRepository

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: flux-manifests
  namespace: flux-system
spec:
  interval: 10m
  url: oci://ghcr.io/fluxcd/flux-manifests
  ref:
    tag: v0.38.2
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: flux
  namespace: flux-system
spec:
  interval: 10m
  path: ./
  prune: true
  wait: true
  sourceRef:
    kind: OCIRepository
    name: flux-manifests
  # patches:
  # patch out wha you want to change

2. GitRepository

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: flux-manifests
  namespace: flux-system
spec:
  interval: 30m
  ref:
    tag: v0.38.2
  url: https://github.com/fluxcd/flux2
  ignore: |
    # exclude all
    /*
    # path to manifests
    !/manifests
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: flux
  namespace: flux-system
spec:
  interval: 30m
  path: ./manifests/install
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-manifests
  # patches:
  # patch out wha you want to change

It's just my personal opinion but I never found the flux bootstrap or flux export commands particularly useful after discovering you can install it with a kustomization.

[kingdonb]

@onedr0p is OCIRepository supported in renovatebot yet?

I think there are some feature gaps. For instance, HelmRepository can be auto-upgraded by renovate, but a HelmRepository with type: oci is not supported yet, last time I looked at least.

Until the OCIRepository ref.tag and similar in GitRepository are covered by renovatebot, then there are still some advantages to the old way. (Hermeticity of the deploy is one advantage, that renovate can keep it up to date is another)

[onedr0p]

@kingdonb OCIRepository is supported I personally submitted the PR however there's a minor bug that needs to be fixed.

GitRepository is now supported and works just fine.

HelmRepository with type oci is not supported yet and if I find time I'll try to submit a PR. This is unrelated to this issue though.

[kingdonb]

What I'm saying is, yes you can patch Flux according to customize Flux manifests but that's not the only way to configure Flux. And users who bootstrapped Flux with custom flags will likely have the expectation that Renovate is not going to wipe out their customizations, (it should ideally behave as if "bootstrapping again with the same parameters")

That's the original subject of this issue, no?

There's a lot to unpack here, I would not recommend anyone do this for example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/fluxcd/flux2/manifests/install?ref=v0.38.2

... because it short-circuits the source controller and will never cache the resource. We recommend people turn off remote resources in Kustomize with no-remote-bases according to the best practices page's guidance. The rest of the advice you gave is perfect, but also not the official way to install Flux at this point. Bootstrap is the official supported way.

So stating that there is a workaround and you don't need support for any more Flux manager custom flags, because users can patch instead, is technically true but also asks users to go off the "garden path" at this point. I would prefer to tell users to bootstrap with --network-policy=false before I would show them how to write a $patch: delete that can accomplish the same thing, it's just less likely to get wrong or be misunderstood.

The other examples you posted are excellent and I'm only picking this minor nit, because I have directed someone at your post who was looking for an example of exactly this, how to do "gitless Flux" and I don't want them to get the wrong idea ;)

Thanks for all the work you're doing here and at-large for Flux!

[onedr0p]

@kingdonb can you expand on this a bit more?

... because it short-circuits the source controller and will never cache the resource.

It is my understanding it is just applying all the resources I need to complete my bootstrap method, this is not being actively watched by flux. I know this isn't the advised route but here is how I am bootstrapping my home Kubernetes cluster, at my day job we do something similar and it works out quite good.

https://github.com/onedr0p/home-ops/tree/main/kubernetes/bootstrap

[kingdonb]

Sure! What I'm saying is, Source Controller handles all of the "source concern" topics like caching, secure provenance, cryptographic verification, or SSH key handling, and Kustomize controller handles all of the "applier concern" topics like kustomize build and runtime decryption of encrypted secrets, and so on.

When you use a remote resource like github.com/fluxcd/flux2/manifests/install?ref=v0.38.2 in your kustomization.yaml it gets evaluated at apply time. So the source archive (artifact) hosted on source controller does not contain the manifests, it only contains that thin reference to the remote that hosts them. If that gets evaluated at apply time, then it's done through the Kustomize CLI (library) and it happens without the benefit of any of the special treatment that source concern topics get in source controller, like caching.

So if you avoid remote bases such as this one, the Source Controller reconciles every GitRepository.spec.interval and it maintains a cache of what it finds there. If next round of reconciling nothing has changed, GitRepository is smart enough to not download the whole git repository again. But if you use the remote base, the Kustomize controller reconciles every Kustomization.spec.interval and it downloads the whole git repository every time (hopefully as a shallow clone, but I honestly have no idea about that). Ideally Kustomize Controller never has to go out and fetch, except from Source Controller.

If you install the Flux monitoring dashboards and check out the Flux metrics, you will absolutely notice a large performance difference between these two methods. Which is why we decided to make the administrative option that prohibits remote bases, since as a cluster admin you might even want to prohibit your users from doing this altogether, as it can have an impact on performance for the whole cluster. I have done this in demos before, but I try not to do that anymore.

In practice, this is probably only an issue at the boundaries of scale. Flux is high performance software and if you don't have hundreds of tenants, you are unlikely to notice scale issues in any case, but what else is most important about this is that certain cloud providers charge for egress, Git repos are often hosted somewhere else. If you are downloading a git repo every 10m versus hitting an endpoint to check if it has the same content it did before by hash, that adds up fast, and you'll potentially see a major impact on your bandwidth bills by making this change, if your bandwidth usage is metered like this.

[onedr0p]

I should have been more clear, the kustomization.yaml that installs flux via the remote ref is only being applied manually, one time and not something flux tracks.

See my bootstrap process here
https://github.com/onedr0p/home-ops/tree/main/kubernetes/bootstrap

[kingdonb]

That's very unconventional and I also like it 🥇

[onedr0p]

I realize it takes a bit more understanding of Flux to do something like this but overall I'm happy with this method.

[Kampfmoehre]

We have a repository with flux components and Renovate did update Flux from 0.39.0 to 0.40.2. The update itself is okay but the commit updated all cluster domains to the default cluster.local overriding our custom cluster domain which broke our flux installation.

We bootstrapped flux using the --cluster-domain argument to set a custom cluster domain. I can see in the commit diff, that the domain is changed like

-       - --events-addr=http://notification-controller.flux-system.svc.custom-domain.local./
+       - --events-addr=http://notification-controller.flux-system.svc.cluster.local./

-       - --storage-adv-addr=source-controller.$(RUNTIME_NAMESPACE).svc.custom-domain.local.
+       - --storage-adv-addr=source-controller.$(RUNTIME_NAMESPACE).svc.cluster.local.

Is there any possibility for Renovate to not update the cluster domain? And does this belong here or should I open a bug report?

[rarkins]

@Kampfmoehre this is not a bug

[HaveFun83]

@Kampfmoehre this is not a bug

But it completely destroyed our flux installation, so there is currently no way to prevent this apart from manually editing the Renovate MR?

I think this issue is not related to renovate itself.
You can workaround this via a custome kustomization file which patch the correct flag or what ever is different to the basic gotk-components.yaml. Example kustomization.yaml

Ping me if you need further help

[mecseid]

@rarkins
I think the namespace specified in the existing gotk-components should be tolerated too.
With the --namespace option you can specify the namespace where you want to install the Flux system.

$ flux install --help
...
Examples:
  # Install the latest version in the flux-system namespace
  flux install --namespace=flux-system
...
Global Flags:
  -n, --namespace string               If present, the namespace scope for this CLI request (default "flux-system")

[kingdonb]

The manager maintainers seemed to be taking a principled stance that only configuration directives that are expressly stated as configuration should be re-parsed. So I don't know if such a change would be accepted, or advisable, but I think the metadata field and way that namespace is configured in Kubernetes API should be considered as stable, so I'm not sure it's worth it to fret over this at all.

It is possible that someone will edit their gotk-components.yaml even though it warns users "DO NOT EDIT" at the top of the file, so how much do you parse? I would look at the Namespace definition itself, and balk if it doesn't match the other resources as expected. That should be safe and stable. What do you think about this?

[mecseid]

Totally understand that only configuration directives should be re-parsed.
The Flux documentation itself says that if you want to change anything (besides bootstrap arguments) then use the kustomization provided strategy merge or JSON patch.

Because of these, it would be nice to support the custom namespace (not flux-system) as you can provide it as a bootstrap parameter and configure it without modifying the generated gotk-components.yaml file (because it will generate it with the appropriate namespace 😊).

[kingdonb]

I didn't even think about that, you know, gotk-components.yaml could say one namespace and then kustomization.yaml overrides it with another.

In Flux docs, we typically tell users to avoid bootstrap configuration flags unless they're really needed, and to put their customizations in the Kustomize overlay for flux-system. Like for example, instead of setting network-policy=false, I'll delete the NetworkPolicy resources with a $patch: delete.

https://fluxcd.io/flux/installation/configuration/boostrap-customization/

https://github.com/kingdonb/csh-flux/blob/dca03d73376690c0eddece911b28895450e5b6bb/clusters/home-workers/flux-system/kustomization.yaml#L28-L50 - (note that patchesStrategicMerge is deprecated and removed in the latest Kustomize versions, I'll have to update this config to use patches: instead... I wonder if it still supports delete?)

If you use this method (not necessary to write a patch, the Kustomization overlay accepts namespace: at the top level and uses it to override resources) then you will not need any bootstrap flags, or need for additional configuration parsers to cajole Renovate into preserving the bootstrap flags' effects.

@mecseid Does this help? I'm not sure if there are embedded namespace references in the Flux manifests that would not be properly set by the Kustomization overlay's namespace: override.