Reduce response payload size of /dag_stats and /task_stats

[XD-DENG]

Their response payload of /dag_stats and /task_stats are like

{
    "example_short_circuit_operator": [
        {
            "state": "success",
            "count": 0,
            "dag_id": "example_short_circuit_operator",
            "color": "green"
        },
        {
            "state": "running",
            "count": 0,
            "dag_id": "example_short_circuit_operator",
            "color": "lime"
        },
        ...
    ],
...
}

The dag_id is already present as the key, but still repeatedly appear in each element,
which makes the response payload size unnecessarily bigger.

In one of the tests I did, this change reduces the response payload size of /task_stats to 23Kb from 33Kb.

---

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[XD-DENG]

I tested and UI is working properly after applying this change.

Another duplicated information in the response payload is the color. It's repeated unnecessarily in each payload, also across payloads. But more broad change is required to address it so I will not touch it for now in this PR.

[BasPH]

Should we write a note in UPDATING.md? Because it's public facing and some people might be using it for whatever reason?

[XD-DENG]

Hi @BasPH , thanks for the reminding.

Earlier I was thinking that it's not directly used by users (only called by UI code) so no need to update UPDATING.md. But you are right, there may be folks doing some hacks using it.

Will add a commit here to update it.

[kaxil]

Can we add a test for this?

[XD-DENG]

Hi @BasPH and @kaxil , updated as suggested.

There are failure in tests CI Build / Requirements (3.6) (pull_request) and CI Build / Requirements (3.7) (pull_request) , and the reasons are not very clear to me (they succeeded in my first commit, and my two following commits don't seem to change anything significant).

May you please help advise? Sorry I'm not very familiar with current CI setup 😅

[potiuk]

Hey @XD-DENG -> It is an interesting thing and it should not happen. I will take a look at that (and likely will have to fix something). This enum34 appearing in generate-requirements should not be there. I will let you know when i fix it (looking at it now.).

[XD-DENG]

Thanks @potiuk ! Appreciate it

[zhongjiajie]

LGTM 👍

[kaxil]

There is a test failure about JSON response too:

>       self.assertEqual(set(resp.json().items()[0][1][0].keys()),
                         {'state', 'count', 'color'})
E       TypeError: 'dict' object is not callable

[XD-DENG]

@potiuk are you fine that I go ahead to merge this first? The latest commit only fails in the two CI Build / Requirements builds.

@kaxil I think I also need to prepare a PR for the flask-admin based non-RBAC UI. Shall I raise it against v1-10-stable directly?

[kaxil]

@potiuk are you fine that I go ahead to merge this first? The latest commit only fails in the two CI Build / Requirements builds.

@kaxil I think I also need to prepare a PR for the flask-admin based non-RBAC UI. Shall I raise it against v1-10-stable directly?

Raise it against v1-10-test please

[XD-DENG]

Hi @potiuk , I have noticed your latest CI fix. Have rebased here.