flux bootstrap git not completing on large repositories

[gpl]

Describe the bug

When attempting to use the following bootstrap command in a large repo, the command seems to never complete. We think it's crashing internally; because it seems to be consistently pulling data from Git (exceeding the size of the repo) - but never completing.

flux bootstrap git --url=ssh://git@github.com/our/repo--branch=trunk --path=clusters/test --private-key-file=/path/to/our/deploy/key --context our-cluster --interval 20m --verbose

Steps to reproduce

Expected behavior

Screenshots and recordings

No response

OS / Distro

Ubuntu 22.04

Flux version

v0.40.2

Flux check

► checking prerequisites
✔ Kubernetes 1.25.3 >=1.20.6-0
► checking controllers
✗ no controllers found in the 'flux-system' namespace with the label selector 'app.kubernetes.io/part-of=flux'
► checking crds
✗ no crds found with the label selector 'app.kubernetes.io/part-of=flux'
✗ check failed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[kingdonb]

Flux is made for one specific purpose, to apply YAML to Kubernetes clusters, and YAML files are small. Typically we recommend bootstrapping Flux into a new repository, rather than into your app repo, and moving configs into the new repo (rather than dragging along all of your legacy commits and non-YAML content with you into the place where you're applying config to the cluster from.) This is my first advice, to get that out of the way. You will have a much better experience with Flux if you use it with smaller Git repos, or with a specific "config repo" that is just for YAML.

That being said, there are several timeout parameters that you can tune to see if it helps. The one thing that will probably not help, is passing --timeout directly to bootstrap git. This is the global context timeout parameter, it does not affect the definition of the GitRepository that Bootstrap produces in any way.

To accomplish this, you would need to "customize Flux resources" as explained here in the FAQ:
https://fluxcd.io/flux/installation/#customize-flux-manifests

The patch you need to apply to Flux is going to look like this one:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: # manifests generated during bootstrap
  - gotk-components.yaml
  - gotk-sync.yaml

patches: # customize the manifests during bootstrap
  - target:
      kind: GitRepository
      name: flux-system
    patch: |
      apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: GitRepository
      metadata:
        name: flux-system
        namespace: flux-system
      spec:
        timeout: 10m0s

and the idea is just to give this clone a little longer, because it is large and expectedly takes so long. The default is 5m.

I do not recommend you move ahead with this configuration, but if you have some operational requirement that the changes must come from this git repo, there are some other options to discuss. (You could pack the important parts for Flux in an OCI repo, and then pull them as an artifact which will again be much more efficient, because only YAML files and some metadata are downloaded.)

Please let us know if any of this information helps, and if there's anything else we can do here!

[stefanprodan]

Flux bootstrap is not made for being run on app repos. Please consider having a dedicated repo for continuous delivery, more on repo structure can be found here: https://fluxcd.io/flux/guides/repository-structure/

[gpl]

Unfortunately, with the bootstrap command, we're not getting to the point where flux is even running on our clusters - the command stalls and seems to constantly crash midway on attempting to pull the repository. It doesn't ever complete. I suspect it's either timing out or crashing internally since it seems to be pulling 100x the amount of data our repo has with a process id that seems to change every handful of seconds; but I haven't had the time to trace or debug.

I appreciate the recommendation, but we have a lot of investment in our monorepo and tooling. We wouldn't be willing to change our repository management framework at this time. It is unfortunate that flux bootstrap is not compatible with this approach; but we were able to write an approximation of the flux bootstrap command that works for our own purposes.

After flux is bootstrapped, flux itself seems to work well and has no problems with our repository, just was a bit harder to bootstrap.

[kingdonb]

It's my recommendation that you do investigate our OCI Repositories, which do not change from Git Repository but are a nice compliment to the Git repo, with traceability back to the commit that created any given tagged OCIRepository Artifact:

https://fluxcd.io/flux/cheatsheets/bootstrap/

This is a new feature, and as the documentation is still in evolutionary state it's unlikely to be marked as GA near as soon as the Git Repository and Kustomization. I am telling you this to help advertise a new feature, I am glad you found a way to make Flux work directly with your Git monorepo after the bootstrap nonetheless!

I think this friction in the bootstrap process of larger repos can be resolved, but I think we're also conscious about not adding too many knobs to bootstrap and other Flux commands. Having multiple timeout commands on a CLI is a recipe for confusion, (but is it worse than having inaccessible timeout values? I'm not sure if this friction is possibly beneficial at all.)

The OCI Repository is something that could be made to work with your setup, it's an additive solution and it is likely to create a measurable benefit in the sense that Flux changes should be able to deploy faster.

If you do stick with Git Repository, I'm sure that the benchmarks and prometheus graphs will be very interesting. Hopefully there is enough monitoring to give you the confidence that Flux is able to handle even large git repos, or at least to give you an idea of when it's really getting too large. I'd be interested in those numbers myself if you have the Grafana dashboards installed. (Thanks for reporting here!)