Airflow gitSync with Submodules

[niraj8241]

I am part of a platform team and the users of the platform have dags in their own repo and org. I have planned to use a single repo with submodule added as part of the onboarding process. I have deployed airflow in Kubernetes using the latest helm chart 1.19.0 and below is my gitsync config

dags:
  persistence:
    enabled: false
    existingClaim: airflow-dags
  gitSync:
    enabled: true
    repo: git@xxxxxxxxxxxxxxxxxxxxxxxxxxxx.git
    branch: master
    subPath: ""
    sshKeySecret: airflow-gitsync
    ref: HEAD
    env:
      - name: GIT_SYNC_SUBMODULES
        value: 'recursive'
      - name: GITSYNC_SUBMODULES
        value: 'recursive'

What I expect to happen:

1. Users keep updating their individual repos
2. git-sync checkouts the sync repo and sub-modules and update the sub-modules to be the latest and sync the changes.

What is Not Working:

1. git-sync is not updating the submodule and I have to run git submodule update --remote --recursive from the parent repo and push it to git and then it works fine.

Any thoughts or suggestions on this behavior?

[potiuk]

Yes. It is expected behaviour of submodules. There needs to be somoene who decides that the current "main" versions of the submodules are bound together in the main repo.

[potiuk]

Could be automated - with web hooks, or polling in the way that you usually automate those things.