DAG Param incorrectly convert enum objects to strings

[cdabella]

Apache Airflow version

2.10.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Using a DAG Param with a type of object and enum set to a list of possible objects is reflected as a string in Trigger DAG UI, both in the selector and in the underlying JSON.

When attempting to run the DAG using the value, the DAG will fail to run

What you think should happen instead?

JSON schemas support enums of any type

Elements in the array might be of any type, including null.

(source: https://json-schema.org/draft/2020-12/json-schema-validation#section-6.1.2-3) . They should be configurable and supported by DAG Parameters, and should allow for successful selection and execution of the DAG from the Trigger DAG UI.

How to reproduce

Create the following DAG

from airflow.decorators import dag, task
from pendulum import datetime
from airflow.models.param import Param

keys = ["a", "b", "c"]
values_display_list = [{"key": key, "value": key.upper()} for key in keys]


@dag(
    start_date=datetime(2024, 1, 1),
    schedule=None,
    catchup=False,
    params={
        "x": Param(
            values_display_list[0],
            type=["object"],
            enum=values_display_list,
        ),
    },
    tags=["example"],
)
def example_params():
    @task
    def example_params_task(*, params, **kwargs) -> None:
        if params:
            print(f"Params: {params}")
        else:
            print("No Params")

    example_params_task()


# Instantiate the DAG
example_params()

Operating System

Debian 12 (Bookworm)

Versions of Apache Airflow Providers

apache-airflow==2.10.1+astro.1

Deployment

Other Docker-based deployment

Deployment details

Astro CLI running Astro Runtime 12.1.0

Anything else?

I think this issue might be caused by the the trigger templating at https://github.com/apache/airflow/blob/2.10.2/airflow/www/templates/airflow/trigger.html#L77-L86

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jscheffl]

Yes, this described problem is a limitatio of the trigger UI form. Object element enums are not support as nobody implemented / requested this feature.

Objects are currently only supported in the UI as JSON entry field. Enums only with strings.

Nevertheless if you trigger via API or manually adjust the generated JSON in the trigger form, JSON validation will work.

Are you willing to supply an improvement?

[cdabella]

I'm willing to try, though front-end work is definitely not my strength.

From a persona perspective, the primary use case for this is when the DAG Author and DAG operator are different users, and the DAG author explicitly is controlling the available actions for their downstream DAG operator users. In this scenario, the DAG operator may not be aware of the underlying DAG code.

I've been playing around with how the trigger.html and trigger.js interact, and specifically the user experience for complex objects that should be considered read-only or selectable from an finite list. This has raised some questions for me.

How to select values?

For standard object params, a CodeMirror text block is added in the parameter form.

Conceptually this doesn't align with our goal of an enum, as the textbox should be read-only, but the question becomes how to select from many read-only text boxes.

Providing a key seems a appropriate. This would need to be the inverse of the existing values_display where the key is actually the parameter value and the value is the UX to be displayed, as complex objects may not be easily serializable into dict keys. I have a working example by adding a keys_display parameter to the Param object, but this could probably be renamed to enum_keys or similar for simplicity.

How to inform users of the selection?

If we are using keyed selection, what is the appropriate way to inform a user of the actual reflection of that selection in the parameters? There is the Generated Configuration JSON view, but this is:

1. Collapsed by default
2. Mutable

We could add a CodeMirror textarea with readOnly set below the enum selector, but this has the potential to greatly expand the form size. Should a view like this always be visible? Collapsable?

It feels like there should maybe be a requirement with the enum param to have an enum_keys param (or autogenerate one).

How to store the values?

Today the values for enums are stored as value attributes on the option tag. For complex JSON objects this leads either to malformed HTML or JSON parse errors because of single quotes. The object could be stored as a child of the option in a script tag, something like the following?

<script id="element_x_option_1" type="text/json">{"key":"a","value":"a"}</script>

Is there a better UX that achieves similar results?

From the use case I'm basing this functionality off of, it feels like there is a more complex ask here around have a Parameter Group where parameters are inter-linked, and must be set as a group. This is beyond my skillset today but I just want to callout that another option is possibly viable for the future.

[jscheffl]

Thanks for the write-up. I assume this level of complexity might also be the reason why this feature was not initially contributed (by me).

Note that the UI is currently under re-work. The current Airflow 2.10-line will not receive additional features. A new ReactJS based UI with a new trigger form is to be developed. So most probably it makes sense to focus on Airflow 3 for this limitation.

Have you considered the alternative to abstract the problem away in the UI such that the parameter selection is a "simple" drop-down, even with values-display which gives an abstract selection in human readable manner to the user. Based on the selection a first task in the DAG resolves the "simple string" into a more complex dictionary with a lookup? That would not need UI complexity and you can hide all technical details in the DAG code.

Something like:

@dag(
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    params={
        "pick_with_label": Param(
            3,
            type="number",
            title="Select one Number",
            description="With drop down selections you can also have nice display labels for the values.",
            enum=[*range(1, 10)],
            values_display={
                1: "One",
                2: "Two",
                3: "Three",
                4: "Four - is like you take three and get one for free!",
                5: "Five",
                6: "Six",
                7: "Seven",
                8: "Eight",
                9: "Nine",
            },
        ),
    },
)
def example_python_decorator():
    @task
    def select_config_details(**kwargs):
        """Select further config parameters depending on user selection."""
        params: ParamsDict = kwargs["params"]
        pick_with_label = params["pick_with_label"]
        my_complex_config = {
            1: {
                "foo": "bar",
            },
            ...
        }
        return my_complex_config[pick_with_label]

Besides all the options... from UX Perspective it might be best to have a MD per option and a complex JSON object and render a radio box selection per option in the enum? Would you expect the user wanting to see/understand the object really or would a description be more appropriate?