Airflow variables and connection management: review and moving forward

[sarayourfriend]

At least twice the topic of how best to manage Airflow variables has come up in conversations between me, @stacimc, and @AetherUnbound. Most recently, Staci brought it up in this comment thread on a PR introducing a new variable:

We could make an infrastructure change to set it as an environment variable though; actually, I'm not clear how we've drawn the line between which variables should be configured where 🤔 What do you think?

­­— @stacimc

To understand Staci's question, and the general question I'm wanting to resolve in this discussion, recall that Airflow has three distinct methods for managing variables. In order of precedence, they are (order retrieved from this Astronomer.io forum thread):

• Secrets Backend
• Environment Variable
• Airflow UI/Metadata Database

We do not have a secrets backend defined, so the two options available to us are environment variables and the Airflow UI.

In previous discussions, we've decided to use the metastore, configured through the UI (private infrastructure repository issue). That may well be the best path forward still. However, I wanted to revisit this (the "through the UI" part of that) in light of Staci's question, and because the previous decision was made before we had set up Ansible as a way of interacting with the Airflow API.

---

A bit of background to set the stage... feel free to skip this if you understand why using a UI to manage application configuration is not always ideal.

Generally speaking, we prefer to configure the environment variables of our services in code. That is part of our Infrastructure as Code (IaC) practice. The primary motivators for IaC for environment variables are:

• to provide a history of changes to variables over time
• to ensure the motivation for changes are documented (e.g., through commit messages)
• to provide visibility of changes, including sometimes to provide opportunity for review of changes by at least one person before they are applied¹
• to provide a standard mechanism for auditable changes to variables, as sometimes required for complete rotation of secrets (in other words, it's harder to check whether something's been rotated in a UI compared to in code)

We use IaC to manage all environment variables/application (or runtime) configuration for the following services:

• Openverse frontend
• Openverse API
• Kibana
• Elasticsearch nodes
• Grafana PDC
• CI/CD environments (GitHub Variables/Secrets)

For Kibana and Elasticsearch, we manage a combination of environment variables and configuration files, so there is precedence and sometimes necessity of keeping a combination of both for various reasons.

We use IaC to manage some variables for Airflow, but not all. Of the methods listed above available for Airflow variable management, we only use environment variables for our IaC managed variables. We manage the following as environment variables for Airflow in our Airflow Ansible role:

• Airflow settings that must be configured using environment variables or an Airflow configuration file
• Some but not all connection strings, including database connections, Slack endpoints, AWS connections
• Some things we've introduced as environment variables specifically, rather than Airflow variables (search for getenv in the catalogue code to find these). I've open an issue to address these.

Everything else, including API keys for providers we ingest from, and more recent additions of configuration settings that may have previous gone into environment-variable only things, are all managed through the Airflow UI. The rest of this discussion should assume we've written our code (or updated it in #4863) so that we can always and reliably retrieve variables from the metastore.

---

Given all three methods of variable configuration can be retrieved through the Metastore Variable.get and related APIs, we can still decide how best to manage variables.

As noted above we have already decided to use the metastore to configure Airflow Variables, and I don't think we need to revisit that discussion. That is, we certainly do not need to move variables towards configuration only via the environment (the opposite of #4863). Additionally, we do not need to invest in the complexity of the remote Secrets Backends now (regardless of if ever).

Instead, I propose we continue to use the Metastore (in line with previous decisions), but start managing them using the Airflow REST API, via our Ansible role for Airflow.

The Airflow REST API exposes endpoints for managing variables and connections:

• Variables (GET, POST): https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_variables
• Connections (GET, POST): https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_connections

Using these endpoints, we can augment our existing Ansible role with the ability to sync variables and connections, even if no other changes exist, and bypass the Airflow restart step in these cases.

We would define connections and variables in the Airflow secrets group_vars as a YAML structure, mirroring the REST API's structure. For example, the SILENCED_SLACK_NOTIFICATIONS variable would be defined like so in group_vars.

airflow_variables:
    - key: SILENCED_SLACK_NOTIFICATIONS
      value: |
        {}
      description: |
        DAG configuration overrides for the provider ingestion workflows. Currently only supports overriding the execution timeout for certain tasks, but allows dynamic overrides at DAG run time. Mapping of DAG ID to a list of dictionaries containing the following keys: "task_id" (a regex pattern that matches the task_id of the task to be modified), and "timeout" (str in "%d:%H:%M:%S" format giving the amount of time the task may take, example: 6d:10h:30m).
        
        Declaration: https://github.com/WordPress/openverse/blob/2cffcb9f8da6961e84a00854a3cd472fd0f9dad8/catalog/dags/providers/provider_workflows.py#L42-L58
        
        Example:
        
        {
            "brooklyn_museum_workflow": [
                {
                    "task_id_pattern": "pull_image_data",
                    "timeout": "10h"
                }
            ]
        }

    - key: ...
      value: ...
      description: ...

Connections would look like this:

airflow_connections:
    - connection_id: ELASTICSEARCH_STAGING
      conn_type: Elasticsearch
      description: etc
      host: elasticsearch.internal
      port: 9200
    
    - connection_id: GITHUB_API
      conn_type: HTTP
      schema: https
      host: api.github.com

Because these YAML structures mirror the REST API's interfaces for variables/connections, it is trivial to export our current configuration to this format.

The role tasks would go something like this:

1. Read the variables and connections from the Ansible role variables, sort them by identifier, and replace sensitive values with sha256 hashes
2. Pull the variables and connections from the API, and repeat the same sorting and hashing process from the previous step, but make sure to save the clear-text versions as well, as we need them later when interacting with the API.
3. Generate a diff of the two and output it for confirmation.
4. If confirmed, split variables/connections into two groups for each type representing the upstream operations: one group for update/create and another for delete.
5. Iterate through the update/create variable group, and issue a POST request to /api/v1/variables with the variable. The POST request works both for updating an existing variable and to create a new one (provided you always supply the entire body of the POST request, which we will).
6. Iterate through the delete variable group, and issue a DELETE request to /api/v1/variables/{{ variable.key }} for each one.
7. Do the same iteration of update/create group and delete group for connections.

---

After this, the answer to Staci's original question will be: always manage variables using the Ansible definitions.

@WordPress/openverse-catalog what do y'all think of this? It's an attempt to bring the management of Airflow configuration/secrets, etc, closer in line with the rest of our applications.

Footnotes

1. Though, whether a change to the environment variables of a service requires review before applying them is up to the Opener making or proposing the change to decide. In some situations (i.e., after a discussion), we might apply changes before review. In others, there is an emergent situation requiring changes from an SRE perspective. Which is to say: review is a notable feature of, but not always a necessary aspect of, environment variable management. Mostly the goal is visibility before or after the fact, irrespective of whether a critical review is required. ↩

[AetherUnbound]

Thanks for the large degree of context here, I'll provide my thoughts when I have time!

[stacimc]

Thanks for starting this discussion, and linking the previous ones; my question came from knowing that variables set via the Admin UI take lower precedence than environment variables (the opposite of how we'd prefer it to work), and recalling that we had discussed how to handle that in the past but not remembering what our conclusion was!

After this, the answer to Staci's original question will be: always manage variables using the Ansible definitions.

If I'm understanding correctly that the proposal here is that we only make changes to Airflow variables using the Ansible definitions, ie never through the Admin UI, I have some strong reservations. Particularly for variables like the SILENCED_SLACK_NOTIFICATIONS and SKIPPED_INGESTION_ERRORS, these are updated fairly frequently. We also use Airflow variables to control the progress of some DAGs (like the batched_update) and it is sometimes necessary to quickly manipulate them. Management through the Ansible definitions adds a layer of complexity that worries me, particularly considering issues we've had with the stability of playbooks.

I agree that maintaining an approval process and history in code for some variables (Airflow connections, contact email, etc) is important. Some of our variables (again, SILENCED_SLACK_NOTIFICATIONS and SKIPPED_INGESTION_ERRORS come to mind) are used for day-to-day DAG management and I don't think require the same level of maintenance; I would argue this is similar to other DAG management tools that are built into Airflow directly, like enabling/disabling DAGs, triggering manual DagRuns, and manually passing/clearing tasks. Silenced/skipped errors and notifications are also already required to have GitHub issues associated with them.

[sarayourfriend]

Thank you, that's great context to have! I definitely did not think of the conflict between my proposed approach and using the metastore variables to maintain DAG state for the likes of the batched update.

I originally considered a hybrid approach, where the Ansible playbook would only update/create, but never delete. I considered this because I wanted deletes to be a special process that went through the UI, considering the importance of Airflow variables. I decided against that because it felt potentially messy, but it sounds like it's actually necessary. That would allow keeping variables that are defined and managed in other ways, whether manually or programmatically in DAG code. The change to the proposed playbook flow above would be to just remove the delete process and only do the create/update process.

However, a mixed approach does make it messy, no matter what, and sends mixed signals about what to do when and why, which would require good documentation and processes. It might present complications with making everyone aware of this process, compared to other applications where there is a single, clear approach.

Just using the UI and nothing else, and ignoring considerations of IaC considerations for these, is an option.

---

On the topic of playbook complexity... (bit of an aside)

I'd argue the complexity of the Airflow Ansible playbook can be significantly reduced by switching Airflow to use the immutable EC2 service approach. That would bring deploying release updates to it in line with our other EC2-based services (aside from ES nodes), which we've been able to deploy repeatedly and successfully. It can also bring Airflow much closer to the ECS services deployment methods if I invest more time into the GitHub-workflow based deployment process for EC2 instances (potentially identical in the overall process). I haven't suggested or pushed for this though because I don't want to create any turbulence with the Airflow deployment process, knowing it's been bumpy going already. Plus, it requires #4648 to be sorted out first anyway. I think this would be an improvement, but I also understand feeling sceptical of that in light of how it's gone so far otherwise. That's just an aside, I don't think a change here is strictly necessary to address the sense of complexity of using Ansible to manage this. Even if we changed the deployment method to the immutable EC2 service approach, we can still use Ansible to interact with the Airflow API or CLI to accomplish IaC approaches to things.

To the point of the existing playbook's complexity and instability: we can separate the variable/connections management into a separate playbook altogether, which bypasses the more complex parts of the role, and would be easier to test and maintain regularly. I'd also put forward that by using Ansible for more of these interactions with Airflow, we would quickly refine the playbooks and stabilise them. Part of the issue we've had with the Elasticsearch and Airflow playbooks, I feel, has come from not using them as regularly as we do the tools for other applications.

---

Clarity of where and which variables to manage, given a mix of approaches is unavoidable with our current implementation

To the point of keeping a clear message of where to configure variables, would a simpler, more confident feeling playbook, open you up to considering managing things like SKIPPED_INGESTION_ERRORS in IaC? That is, with the understanding that you would be able to apply those changes without needing to wait for review first? As clarified in my overview, we don't always need to or can wait for review when changing application configuration, so it's built in that you would be able and allowed to do dynamic and quick changes. The PR would be a formality to ensure documentation, history, clarity of purpose, etc (and sometimes just a routine sign-off), similar in scope to the changelog PRs we have today.

Provided the playbook to apply changes was quick and confident, managing variables/connections with it should (I would think) be as fast and hopefully more comfortable as logging into the Airflow UI and editing the JSON blob in the textarea. But you have to tell me whether that is true, it's just my perception and hunch.

That does not address the fact that some metastore variables are managed as part of DAG flows and state tracking. Leaving aside that I don't know whether that's the best way of tracking what we are tracking (I'll trust that it is better to do that than using a for-purpose Postgres table), we'd still have to communicate the fact that those variables are distinct in some way from the ones that get managed in IaC. There is still a messaging difficulty, which gives me pause to the idea of IaC being able to be applied and understood consistently in this area.

---

Alternative approaches than Ansible

Lastly, if Ansible is the primary difficulty or road block to using IaC to manage, there is at least one alternative I can think of, short of adopting one of the Secrets Backends.

We could use a DAG to manage these variables, and have it pull the definitions from S3, and then manage the file in S3 using Terraform (with which we are all familiar). We wouldn't use Ansible at all, in that version, but still be able to define variables with a familiar IaC approach. They would just come with whatever delay the DAG's schedule was on. It's a small chunk of work to do, so maybe it could be an every-minute DAG? In which case the delay would not be meaningful in most cases, I suppose especially those where we wouldn't stick to using the UI to manage them anyway.

We could also adopt a Secrets Backend, which would have similar ergonomics (configure values in Terraform), but without needing to write our own glue code to make it work.

---

Your response and thinking further about this has clarified to me that the question is less about "what is right, in light of IaC", and more: what is best for the folks interacting with Airflow variables on a regular basis, and what will make y'all feel most confident and comfortable when doing that. Are the potential benefits of an IaC approach attractive such that exploring this topic is valuable (regardless of whether the method is an Ansible playbook, a DAG to glue S3 files, or a Secrets Backend). Ansible is one approach I could think of that was relatively low-lift and didn't change much. But if an IaC approach has value from y'all's perspective, and Ansible is a roadblock to that end, then it's good to make clear that Ansible is not a necessary aspect of it. From the infrastructure side of things, we can enable whatever approach y'all want. And if that's just continuing to use the UI and nothing else because the IaC benefits aren't worth making a change, then great! I don't mind that at all either.

In other words, the goal of this discussion can be two-layered:

1. Foremost: does any IaC approach to managing variables/connections for Airflow make sense and bring valuable benefits to the folks working with Airflow at that level?
2. If that is the case, what are the acceptable approaches to implementing an IaC approach?

We can focus on the first question, because the discussion could end quickly if the answer is "no". I wish I'd started the discussion this way originally, as it feels like a lot clearer process. The second point begs the question of IaC being a net benefit, which maybe it isn't in this case... which would mean we can skip the second question altogether.