Ensure that the the generated airflow.cfg contains a random jwt_secret and fernet_key

[ashb]

I don't know exactly when this got broken, and unforutnaltey it wasn't tested,
but I suspect it might have been around the time we swapped the default config
from a config file to the yaml based values. I.e. a while ago!

To make sure it doesn't get broken I've gone and added some unit tests

And to make my next PR and test easier I have done the same thing with the
auth_jwt_secret that we do for fernet_key -- of only set it in the config
file if we're generating that file, not always in memory.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

PTAL @vincbeck and @pierrejeambrun . I do worry about the upgrade path, but the issue of generating the jwt_signing key in memory was causing me issues in my work on updating the JWT signer to support public/private key in another PR.

What do you think?

[pierrejeambrun]

LGTM

[vincbeck]

Nice!

[vincbeck]

Something I realized with this change. We use the fernet key to encrypt trigger kwargs in the DB. Now we have long running triggers, when you restart your environment the Fernet key is regenerated and thus we receve an exception when trying to deserialize the trigger because the key is not valid. I guess this will not happen in prodution because deployment manager should set the Fernet key but that's something inconvenient for development purposes. I need to reset the DB each time

[ashb]

Something I realized with this change. We use the fernet key to encrypt trigger kwargs in the DB. Now we have long running triggers, when you restart your environment the Fernet key is regenerated and thus we receve an exception when trying to deserialize the trigger because the key is not valid. I guess this will not happen in prodution because deployment manager should set the Fernet key but that's something inconvenient for development purposes. I need to reset the DB each time

This should have been fine I think - the generation happens when writing out the temp my/sample airflow.cfg file for the first time only, it's not in memory each time airflow starts. Is that file not persisted in how ever you are developing?

[potiuk]

Something I realized with this change. We use the fernet key to encrypt trigger kwargs in the DB. Now we have long running triggers, when you restart your environment the Fernet key is regenerated and thus we receve an exception when trying to deserialize the trigger because the key is not valid. I guess this will not happen in prodution because deployment manager should set the Fernet key but that's something inconvenient for development purposes. I need to reset the DB each time

This should have been fine I think - the generation happens when writing out the temp my/sample airflow.cfg file for the first time only, it's not in memory each time airflow starts. Is that file not persisted in how ever you are developing?

I think in breze it's not persisted, but there is nothing preventing to do so. We can add it @vincbeck - if that's what you are referring to - we can create a separate volume for airflow configuration folder (I think currently it's just regular container filesystem - so every time you restart container it's gone). WDYT?

[vincbeck]

Yes that would work nicely :) Yes sorry I did not mention it but I use breeze to develop and that's where I experience the issue

[potiuk]

there it goes @vincbeck #47228

[potiuk]

Unitl you run breeze down - all the configuration and database data is kept which means that you can easily:

• close and restart breeze without loosin fernet key and jwt_secrets
• also easily clean them up by running breeze down

[jscheffl]

Okay, just saw this PR... because came-down attempting to debug why EdgeWorker in breeze fails since a few days.
In breeze if the secrets are not set (they are not per default) then every process call generates a new token. If you start breeze and run airflow config list|grep secret_key you see that there is with every call a new secret being generated.

While I assume this is a good hardening, I think in breeze we need to fix this. Will raise a PR in a moment.

Otherwise, did we consider back-porting to 2.10 branch? If it is security relevant we should bring it there as well?

[github-actions]

Backport failed to create: v2-10-test. View the failure log Run details

Status	Branch	Result
❌	v2-10-test

You can attempt to backport this manually by running:

cherry_picker 106b872 v2-10-test

This should apply the commit to the v2-10-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue