Refactor config manager to use a hierarchical pattern

[MoralCode]

Description

New config entries, such as those added in #3337, #3346, and #3353, currently need to be manually added to the database by users who are using augur deployed using the default, manual instructions in the docs. Previously this was done using database migrations, however this is not best practice and should not be done going forward.

Without these migrations, new config entries need to be manually added to the database by users who are using augur deployed using the manual instructions in the docs. This can cause crashes for users with this kind of setup.

This change proposes a better way to handle configs going forward by refactoring the internals of AugurConfig to implement a more "hierarchical" approach to fetching config values.

With this change, configuration will now be considered from all available sources and will be read from in the following order - with values higher in this list taking priority over those that are lower down:

• The augur database
• the augur.json file (if present in the configured config directory as introduced in Allow default config to be read from a file #3314)
• the hardcoded set of default config values.

This change is accomplished by introducing a ConfigStore class to act as an interface, as well as two implementations of that class, JsonConfig, and DatabaseConfig. These two implementation classes represent different "storage backends" available for storing and retrieving configuration values. This enables the main AugurConfig class to effectively and simply handle various configuration-fetching logic.

Along with these new classes, a new "writable" flag has been introduced, allowing ConfigStore implementations to indicate whether or not they support writing to the configuration backend they represent. This flag is used in the logic of AugurConfig to assist with deciding which backend to write to when operations such as create_section, add_value etc are called. Currently the logic will look for the config store with the highest precedence that is writable and will direct all write operations to that. With the current precedence rules, this means writes are still directed to the config database, however this is not a hardcoded requirement. If write operations are attempted with no configured writable backends, the operations will simply silently return and not be written.

Docker-based installs also call the Augur CLI to initialize the config on startup. The logic for this has also been migrated and works ~90% the same as it used to. Prior to this change, the config init CLI command would grab the default config (which would come from EITHER the augur.json or the hardcoded defaults), inject values from provided environment variables (API keys, connection strings, etc), and then write that config back to the database. With this change, the config init CLI grabs a set of config values built from BOTH augur.json and the hardcoded defaults. This is now known as the "base config" and is assembled from all read-only config sources configured in AugurConfig.

This PR fixes #3356

Testing
Some unit tests have been added to cover the JsonConfig class to ensure that it works. This can be run with uv run tox.
Due to technical hurdles and the requirements to provide a database connection for the DatabaseConfig class, this class is not unit tested, but is built from code snippets from the previous AugurConfig implementation that have already been in production use

I will also be running this on my local instance to test it and I encourage others to do so as well

Notes for Reviewers

I invite discussion around the logic surrounding how to gather different subsets of configuration. Im still thinking through the process, specifically how to determine which config backend to write to and how to determine which backends to combine to represent the "base" config.

also happy to split this change across multiple files if it is preferred to give each class its own file.

Signed commits

• Yes, I signed my commits.

[github-actions]

[pylint] _{reported by reviewdog 🐶}
E1101: Instance of 'DatabaseConfig' has no 'logger' member (no-member)

[github-actions]

[pylint] _{reported by reviewdog 🐶}
E1120: No value for argument 'logger' in constructor call (no-value-for-parameter)

[MoralCode]

Im ignoring the codeql misspell CI failure because that came from the inclusion of the unit testing config, where a folder and its associated job name (which are unused here) are misspelled. Im considering this out of scope for this PR given the already large diff.

I can address this in a followup if it is important

[MoralCode]

I'm also thinking instead of all the complexity surrounding the dynamic filtering of the list of config store backends, It may be easier to just separately store/designate a single config entry that is the writable one and use that. it would be less dynamic, but I think thats acceptable because realistically these config backends aren't likely to change much (other than maybe introducing a backend to interface with the "configmanager" service (think like keyman), once that exists.

[sgoggins]

@MoralCode : Obviously there is a lot new here, but your design rationale is exceptional, and we 100% need to do this. My only questions are:

• Do we have a dev machine we could test this on with ~200 repos just to make sure nothing breaks. I think stopping and starting Augur a couple times will get us to where we need to go. This is not a "hard requirement", just a question.
• I forgot my other question.

I am good with merging this, despite the lengthy comments.

[sgoggins]

@MoralCode : Obviously need to fix the CII ... Do we think we need to have @JohnStrunk revisit our current Docker builds? It is unclear if we have made docker brittle or more brittle, or if there is just some brittleness related to the changes we are making.

[MoralCode]

@MoralCode : Obviously need to fix the CII ... Do we think we need to have @JohnStrunk revisit our current Docker builds? It is unclear if we have made docker brittle or more brittle, or if there is just some brittleness related to the changes we are making.

I don't think we need to pull John in. I have a decent amount of container experience at this point, and I think the dynamic of two required reviews, one from someone using a manual install and one from someone on docker is good, especialy if we intend to support both. My bigger issues are more surrounding docker vs podman differences to be honest.

I'll get the CI working.

[shlokgilda]

There's a typo in tests/test_applicaton. Should be test_application (missing i.) Not sure if if's within the scope of this PR but if you're updating tests/test_applicaton/test_config/test_config.py, is it prudent to fix the typo? Or should I open a separate issue and PR for that?

[sgoggins]

LGTM