Support syncing and updating more than one git repository

[squaremo]

From time to time people say that it would be handy if flux supported syncing with more than one repo.

These are use cases:

• We keep our configs alongside our app code, and that's in one repo per app
• I have a infra layer of config that I only want to sync, and then other app config that I want to sync and update

These are not use cases:

• We have different teams, and they each have a repo with config (use multiple fluxd daemons, so you can narrow the RBAC permissions for each)

Also see #1139.

[rade]

Would git subtrees be of help here? After all, they do allow you to in effect aggregate multiple repos into one.

[squaremo]

Would git subtrees be of help here? After all, they do allow you to in effect aggregate multiple repos into one.

Maybe, but I'm not sure it would be the simplest way to accomplish this. (Genuinely not sure, rather than British not sure)

[Smana]

We're exactly in the first use case and we have multiple repositories.

We keep our configs alongside our app code, and that's in one repo per app

Keeping the "config" with the code allows the devs to work in a single place on the project and on how it is deployed.

Let's suppose you deploy 50 projects in a given namespace and you want to keep the helm charts (that's what we're using) along with the code. Does it mean you'll have 50 operators in the same namespace ? and how about the helm-operator ? I guess there's some work to do if you want to keep a single tiller.

I'm looking for advices on the best setup in my case. I'm going to explore 2 options:

• The first one as explained above, one flux-operator per git repository
• A new repository with a directory tree per namespace and a flux operator per namespace

Help is welcome :)

[Smana]

I managed to have something that fits my needs.

I've create a very simple operator that deploy fluxes :)
In a namespace flux I have:

• 1 tiller
• a flux per github repository
• a helm operator per repository

Todo:

• use a single memcache instance for all fluxes if possible
• limit the scope of the fluxes (RBAC)

[mazzy89]

@Smana what it scares me with that approach is that you need to have a flux for each service that you want to deploy. Basically, you need to double the resources in your cluster just to accommodate flux 😱. If you have 50 microservices, you need to have 50 different flux daemons running. So you will eat your cluster resources faster.

I think at the moment the only feasible approach is to have mono repos.

[Smana]

@mazzy89 I understand your concerns, but I can't change what's has been build for years and the current status of the github repositories in my current.
That said i'm pretty happy with my setup.
Here's an updated version:

• 1 flux operator in charge of creating fluxes
• 1 memcached cluster for all the instances
• 1 sealed-secrets operator

Todo:

• The sealed secrets option is temporary, we're thinking about dynamic secrets injection at pod startup with vault or even security vendor solutions (POCs starting soon)
• Limit the fluxes instances to a single namespace
• We want to keep the decision of changing the image tag in our CI workflow. I'm going to do some tests.

But yes, I agree, we'll have number of couples flux/helm-operator :

flux-operator-5d957f64f7-flsfj                                    1/1     Running   0          2d20h
project1-namespace1-4wpi7jr0p61vonmvidlatf8k-flux-7f444bd5c9-859w9       1/1     Running   0          2d19h
project1-namespace1-4wpi7jr0p61vonmvidlatf8k-flux-helm-operator-65nnh5   1/1     Running   0          2d19h
project2-namespace2-qtldck8q7sj0kgdyo8ixjyw4-flux-545d8c6d48-g627s     1/1     Running   0          2d
project2-namespace2-qtldck8q7sj0kgdyo8ixjyw4-flux-helm-operatorcgspj   1/1     Running   0          2d
project3-namespace3-a59qvo33udy3gu7l5klwjpmck-flux-748657bcf7gwl97   1/1     Running   0          47h
project3-namespace3-a59qvo33udy3gu7l5klwjpmck-flux-helm-operar595g   1/1     Running   0          47h
project4-namespace4-9z7xaxmhodgomfqoxnls85kro-flux-96697b455-hsb5x   1/1     Running   0          41h
project4-namespace4-9z7xaxmhodgomfqoxnls85kro-flux-helm-operag4xpg   1/1     Running   0          41h
project5-namespace5-7k9qgulhls6zfmsevylvex1lw-flux-6d9f76f7lqkkz   1/1     Running   0          43h
project5-namespace5-7k9qgulhls6zfmsevylvex1lw-flux-helm-opehfvm6   1/1     Running   0          43h
memcached-0                                                       1/1     Running   0          3d13h
memcached-1                                                       1/1     Running   0          3d13h
memcached-2                                                       1/1     Running   0          3d13h
tiller-deploy-67b4548b8c-b579k                                    1/1     Running   0          4d12h

[stillinbeta]

My use case is relatively simple: I'd like to have most of my configuration live in an open source repository, and then just have secrets live in a private repository. Multiple repositories seems like the easiest way to accomplish this, but I may be missing something obvious

[cruisehall]

Has anyone seen a pattern for compiling k8s manifests from multiple repositories using something like jsonnet (or the late ksonnet)? https://jsonnet.org/

With features like “flux release”, it certainly seems like flux intends to treat application source code as a first class citizen, but by doing so flux is biting off more than it can (or should?) chew.

If flux is just a tool for managing cluster state, it’s fair to say that flux only needs to be concerned with one repo - the “cluster repo”. Leave it to users to figure out how to populate that repo with their “fully compiled” k8s manifests.

But if flux is a tool for managing cluster state AND releasing applications (among other things),, then it must be able to handle resolving resources from multiple repositories.

[toran414]

We keep our configs side by side the code, each workload in its own repo. We also would like to use GitFlow with different branches in each of those repos applying to different namespaces in our cluster.

[tschonnie]

I am confused. Does https://github.com/fluxcd/multi-tenancy solve this issue or not?

[hiddeco]

I am confused. Does fluxcd/multi-tenancy solve this issue or not?

It does kind of solve this by spawning multiple Flux daemons, this issue is however about supporting multiple git repositories with one Flux daemon, which the mentioned repository does not solve.