Move the database configuration to a new section #15930
Description
Hello.
Currently, the core section has 48 configuration options. This is a bit of a lot and it becomes problematic to find the configuration option that interests us. Our users ask questions about an option that already exists by this.
Below I present the statistics, how many options each section has.
$ yq eval . --tojson airflow/config_templates/config.yml | jq -r 'map({key:.name, value:(.options|length)})|sort_by(.value)|reverse |.[]| [.key, .value]|@tsv'
core 48
webserver 46
scheduler 24
celery 24
kubernetes 21
logging 20
smtp 9
api 9
metrics 9
elasticsearch 7
operators 7
email 6
kerberos 5
atlas 5
smart_sensor 4
dask 4
elasticsearch_configs 2
sentry 2
hive 2
cli 2
secrets 2
github_enterprise 1
celery_broker_transport_options 1
celery_kubernetes_executor 1
lineage 1
debug 1It would be great if we could move some options to the new section. I think we can start with the database-related options that we can move to the new [database] section.
sql_alchemy_conn
sql_engine_encoding
sql_engine_collation_for_ids
sql_alchemy_pool_enabled
sql_alchemy_pool_size
sql_alchemy_max_overflow
sql_alchemy_pool_recycle
sql_alchemy_pool_pre_ping
sql_alchemy_schema
sql_alchemy_connect_args
load_default_connections
max_db_retries
We've made similar changes in the past.
#12165
#6887
I also wonder what other sections we can divide to make the configuration even easier. What do you think that the configuration options for our flask application should be moved to the webui section? This is to separate the infrastructure/webserver from the UI/browser options.