After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

rotemseekingalpha · 2022-05-09T13:37:42Z

Apache Airflow version

2.3.0 (latest released)

What happened

After the upgrade from 2.2.4 to 2.3.0, some DAGs grid data seems missing and it renders the UI blank

What you think should happen instead

When I click the grid for a specific execution date, I expect to be able to click the tasks and view the log, render jinja templating, and clear status

How to reproduce

Run an upgrade from 2.2.4 to 2.3.0 with a huge database (we have ~750 DAGs with a minimum of 10 tasks each).
In addition, we heavily rely on XCom.

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

apache-airflow apache_airflow-2.3.0-py3-none-any.whl
apache-airflow-providers-amazon apache_airflow_providers_amazon-3.3.0-py3-none-any.whl
apache-airflow-providers-ftp apache_airflow_providers_ftp-2.1.2-py3-none-any.whl
apache-airflow-providers-http apache_airflow_providers_http-2.1.2-py3-none-any.whl
apache-airflow-providers-imap apache_airflow_providers_imap-2.2.3-py3-none-any.whl
apache-airflow-providers-mongo apache_airflow_providers_mongo-2.3.3-py3-none-any.whl
apache-airflow-providers-mysql apache_airflow_providers_mysql-2.2.3-py3-none-any.whl
apache-airflow-providers-pagerduty apache_airflow_providers_pagerduty-2.1.3-py3-none-any.whl
apache-airflow-providers-postgres apache_airflow_providers_postgres-4.1.0-py3-none-any.whl
apache-airflow-providers-sendgrid apache_airflow_providers_sendgrid-2.0.4-py3-none-any.whl
apache-airflow-providers-slack apache_airflow_providers_slack-4.2.3-py3-none-any.whl
apache-airflow-providers-sqlite apache_airflow_providers_sqlite-2.1.3-py3-none-any.whl
apache-airflow-providers-ssh apache_airflow_providers_ssh-2.4.3-py3-none-any.whl
apache-airflow-providers-vertica apache_airflow_providers_vertica-2.1.3-py3-none-any.whl

Deployment

Virtualenv installation

Deployment details

Python 3.8.10

Anything else

For the affected DAGs, all the time

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

boring-cyborg · 2022-05-09T13:37:43Z

Thanks for opening your first issue here! Be sure to follow the issue template!

rotemseekingalpha · 2022-05-09T13:45:21Z

Video that shows the problem

jpipas · 2022-05-09T14:24:22Z

Getting the same thing - It appears to be an issue with the getTask API (there's several javascript errors in console). Most notably the /tasks endpoint for a given dag.

Python version: 3.9.12
Airflow version: 2.3.0+astro.3

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/validation.py", line 399, in wrapper
    return function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/response.py", line 112, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/parameter.py", line 116, in wrapper
    return function(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/security.py", line 49, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/endpoints/task_endpoint.py", line 67, in get_tasks
    return task_collection_schema.dump(task_collection)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 765, in _serialize
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 765, in <listcomp>
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 634, in _serialize
    return schema.dump(nested_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 1870, in _serialize
    return self._serialize_method(obj)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/schemas/task_schema.py", line 70, in get_params
    return {k: v.dump() for k, v in params.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/schemas/task_schema.py", line 70, in <dictcomp>
    return {k: v.dump() for k, v in params.items()}
AttributeError: 'dict' object has no attribute 'dump'

bbovenzi · 2022-05-09T15:31:17Z

Hmm thats a new one. @norm would you mind taking a look at get tasks?

jpipas · 2022-05-14T12:11:07Z

Just wanted to update, that this appears to only be happening for DAGs that have TaskGroups. When viewing my "non-taskgroup" DAGs, the grid-view works as expected. When trying to select individual tasks on a DAG with task groups (and therefore have expand/collapsible grid), clicking on an individual task the entire grid view disappears.

ashb · 2022-05-17T20:00:03Z

Can someone give us a DAG and re-production steps please?

bbovenzi · 2022-05-18T16:42:50Z

Also, what are the other js console errors aside from the get task endpoint? Even if that endpoint throws an error, the UI shouldn't crash

rotemseekingalpha · 2022-05-19T11:52:23Z

I get first error message twice when then the second when the Grid view loads:

# First
VM35:1          
GET https://****************/api/v1/dags/some_dag/tasks 500
    (anonymous)                 @   VM35:1
    (anonymous)                 @   tree.ac7202f….js:2
    e.exports                   @   tree.ac7202f….js:2
    e.exports                   @   tree.ac7202f….js:2
    c.request                   @   tree.ac7202f….js:2
    r.forEach.c.<computed>      @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    fetchFn                     @   tree.ac7202f….js:2
    s                           @   tree.ac7202f….js:2
    c                           @   tree.ac7202f….js:2
    t.fetch                     @   tree.ac7202f….js:2
    n.executeFetch              @   tree.ac7202f….js:2
    n.onSubscribe               @   tree.ac7202f….js:2
    t.subscribe                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    Dl                          @   tree.ac7202f….js:2
    t.unstable_runWithPriority  @   tree.ac7202f….js:2
    Go                          @   tree.ac7202f….js:2
    Bl                          @   tree.ac7202f….js:2
    bl                          @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    t.unstable_runWithPriority  @   tree.ac7202f….js:2
    Go                          @   tree.ac7202f….js:2
    Yo                          @   tree.ac7202f….js:2
    Xo                          @   tree.ac7202f….js:2
    xl                          @   tree.ac7202f….js:2
    is                          @   tree.ac7202f….js:2
    t.render                    @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    n                           @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    n                           @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2

# Second
Error: Request failed with status code 500
    at e.exports (tree.ac7202f96db7173e9baf.js:2:141127)
    at e.exports (tree.ac7202f96db7173e9baf.js:2:297309)
    at XMLHttpRequest.E (tree.ac7202f96db7173e9baf.js:2:139518)

And finally, the following error twice when I click one of the tasks:

tree.ac7202f96db7173e9baf.js:2 
TypeError: Cannot read properties of undefined (reading 'tasks')
    at B_ (tree.ac7202f96db7173e9baf.js:2:739128)
    at ui (tree.ac7202f96db7173e9baf.js:2:228656)
    at Qu (tree.ac7202f96db7173e9baf.js:2:280737)
    at Pl (tree.ac7202f96db7173e9baf.js:2:268010)
    at jl (tree.ac7202f96db7173e9baf.js:2:267938)
    at Cl (tree.ac7202f96db7173e9baf.js:2:267801)
    at bl (tree.ac7202f96db7173e9baf.js:2:264788)
    at tree.ac7202f96db7173e9baf.js:2:214575
    at t.unstable_runWithPriority (tree.ac7202f96db7173e9baf.js:2:291432)
    at Go (tree.ac7202f96db7173e9baf.js:2:214352)
cu                          @ tree.ac7202f96db7173e9baf.js:2
n.callback                  @ tree.ac7202f96db7173e9baf.js:2
ma                          @ tree.ac7202f96db7173e9baf.js:2
gu                          @ tree.ac7202f96db7173e9baf.js:2
Il                          @ tree.ac7202f96db7173e9baf.js:2
t.unstable_runWithPriority  @ tree.ac7202f96db7173e9baf.js:2
Go                          @ tree.ac7202f96db7173e9baf.js:2
Tl                          @ tree.ac7202f96db7173e9baf.js:2
bl                          @ tree.ac7202f96db7173e9baf.js:2
(anonymous)                 @ tree.ac7202f96db7173e9baf.js:2
t.unstable_runWithPriority  @ tree.ac7202f96db7173e9baf.js:2
Go                          @ tree.ac7202f96db7173e9baf.js:2
Yo                          @ tree.ac7202f96db7173e9baf.js:2
Xo                          @ tree.ac7202f96db7173e9baf.js:2
Me                          @ tree.ac7202f96db7173e9baf.js:2
Yt                          @ tree.ac7202f96db7173e9baf.js:2

ashb · 2022-05-19T15:19:23Z

Great thanks -- the second error is likely a side effect of the first.

What error appears in the webserver logs when you do the first action?

jpipas · 2022-05-19T16:03:32Z

The web server error is what I posted above... ends with:

AttributeError: 'dict' object has no attribute 'dump'

I too am getting the same JS errors that @rotemseekingalpha is showing as well. My theory about taskgroups however, appears wrong because it's also happening on "non-taskgroup" DAGs. In my case they're "previously parsed" dags, and were there before the upgrade to 2.3.0 - so I assume there's something in the migration for these dags or that happened to them in the past that 2.3.0 just doesn't like.

ashb · 2022-05-19T20:41:05Z

Do you have task.params or dag.params set to by any chance?

If someone can share with me a dag that exhibits this behaviour that would be very useful.

rotemseekingalpha · 2022-05-20T07:12:08Z

@ashb we rely heavily on task.params

ashb · 2022-05-20T12:28:32Z

I'll say it for the third time: we need a DAG that has this behavoiur to reproduce and fix it please.

zachliu · 2022-05-25T15:24:02Z

@ashb steps to reproduce: #23908 which seems to be a more severe version of this issue: the grid view can't be rendered at all

ashb · 2022-05-27T14:53:25Z

@zachliu That grid view issue was caused by something else (removing a task from a DAG after it has run) to the 500 errors reported in this thread.

maf-rnmourao · 2022-06-02T07:15:22Z

The issue seems to happen only when you are not logged in. It can be probably solved by customizing some permission.

Not logged in:

Logged in:

bbovenzi · 2022-06-02T13:16:48Z

Got it. At least one step: We need to make sure an error on /tasks doesn't crash the UI

NilsJPWerner · 2022-06-08T21:49:23Z

I'm having the same issue:

Thank you for your fix @bbovenzi! Looking forward to it in the next version!

junaidnasir-ps · 2022-06-09T15:36:09Z

it's also happening for us when the user role is viewer (not logged in). (for admin role it works fine)
also using helm and k8 with airflow:2.3.2-debian-11-r2 (docker.io/bitnami/airflow:2.3.2-debian-11-r2)

jpipas · 2022-06-13T20:24:15Z

We were able to determine that this issue was caused by a custom operator that erroneously had params as part of its template_fields. Therefore any DAG that contained this operator would "flash" the grid - and then the subsequent API calls would fail - which then caused the grid to disappear. After removing this "keyword" used by the baseoperator from template_fields the grids load properly now.

Before: template_fields = ["sql_query", "s3_key","params"]
After: template_fields = ["sql_query", "s3_key"]

uranusjr · 2022-06-14T04:35:41Z

@jpipas Do you know what error the API is failing with? (May be available in web server logs.) Ideally the web server should be a bit more resilient to this kind of user errors.

IonutArmeanu-work · 2022-06-16T13:35:33Z

I get the same error when I'm using AUTH_ROLE_PUBLIC = 'Admin' in webserver_config.py.
The /tasks endpoint fails with HTTP 401 ( Unauthorized )

j-adamczyk · 2022-06-22T11:45:16Z

I get the same error, but Grid View disappears almost immediately after entering that page, without clicking anything. Also I have small number of DAGs.

kobethuwis · 2022-09-21T10:40:33Z

@cloventt @rotemseekingalpha I'm using the official Airflow helm chart. Encountered the same error. In the chart, we use AUTH_ROLE_PUBLIC = 'Admin' as well as webserver config.

Have been able to fix this error by adding the following to the chart:

extraEnv: |
  - name: AIRFLOW__API__AUTH_BACKENDS
    value: "airflow.api.auth.backend.default"

cgadam · 2022-09-21T18:26:24Z

@cloventt @rotemseekingalpha I'm using the official Airflow helm chart. Encountered the same error. In the chart, we use AUTH_ROLE_PUBLIC = 'Admin' as well as webserver config.

Have been able to fix this error by adding the following to the chart:
extraEnv: |
  - name: AIRFLOW__API__AUTH_BACKENDS
    value: "airflow.api.auth.backend.default"

@kobethuwis thank you for this tip! Adding airflow.api.auth.backend.default to AIRFLOW__API__AUTH_BACKENDS also fixed this issue in my case! Nice workaround until we have this issue solved! Thanks!

radiophysicist · 2023-06-23T10:13:46Z

We have the same problem in some DAGs (others working as expected) after upgrade from 1.0.15 to 2.5.3).
No errors both in browser console and webserver logs

mjkonarski-b · 2023-06-26T16:17:44Z

I believe I found the root cause of the problem.

A new feature was introduced in 2.3.0 - "details drawer" in Grid View (#22123). Under the hood in calls /confirm endpoint here. The problem is that this endpoint requires
CAN_EDIT RESOURCE_DAG and CAN_EDIT RESOURCE_TASK_INSTANCE permissions, which are obviously not granted to the Viewer role by default.

As a result the /confirm XHR call is redirected to the login page and returns its HTML content, which breaks the logic behind the details drawer and we get a blank page.

On top of that #30373 introduced more improvements to the details drawer and now there's also a POST being sent to /clear which requires even more permissions.

EDIT: It's not actually the details drawer itself that is problematic, but its "Mark task as" modals.

potiuk · 2023-06-26T16:58:08Z

cc: @bbovenzi @pierrejeambrun ^^

rotemseekingalpha added area:core kind:bug This is a clearly a bug labels May 9, 2022

bbovenzi added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. and removed area:core labels May 9, 2022

ashb added the pending-response label May 21, 2022

bbovenzi mentioned this issue Jun 3, 2022

Fix useTasks crash on error #24152

Merged

ephraimbuddy added this to the Airflow 2.4.0 milestone Aug 16, 2022

eladkal added the affected_version:2.3 Issues Reported for 2.3 label Aug 16, 2022

ashb modified the milestones: Airflow 2.4.0, Airflow 2.4.1 Sep 8, 2022

jedcunningham modified the milestones: Airflow 2.4.1, Airflow 2.4.2 Sep 27, 2022

ephraimbuddy modified the milestones: Airflow 2.4.2, Airflow 2.4.3 Oct 18, 2022

ephraimbuddy modified the milestones: Airflow 2.4.3, Airflow 2.4.4 Nov 9, 2022

ephraimbuddy modified the milestones: Airflow 2.4.4, Airflow 2.5.0, Airflow 2.5.1 Nov 23, 2022

ephraimbuddy modified the milestones: Airflow 2.5.1, Airflow 2.5.2 Jan 12, 2023

ephraimbuddy modified the milestones: Airflow 2.5.2, Airflow 2.5.3 Mar 10, 2023

pierrejeambrun modified the milestones: Airflow 2.5.3, Airflow 2.5.4 Mar 28, 2023

ephraimbuddy modified the milestones: Airflow 2.5.4, Airflow 2.6.1 Apr 13, 2023

eladkal removed this from the Airflow 2.6.1 milestone Apr 28, 2023

bbovenzi mentioned this issue Aug 1, 2023

Only load task action modal if user can edit #32992

Merged

bbovenzi closed this as completed in #32992 Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

rotemseekingalpha commented May 9, 2022

boring-cyborg bot commented May 9, 2022

rotemseekingalpha commented May 9, 2022

jpipas commented May 9, 2022 •

edited by uranusjr

Loading

bbovenzi commented May 9, 2022 •

edited

Loading

jpipas commented May 14, 2022

ashb commented May 17, 2022

bbovenzi commented May 18, 2022

rotemseekingalpha commented May 19, 2022 •

edited

Loading

ashb commented May 19, 2022

jpipas commented May 19, 2022

ashb commented May 19, 2022

rotemseekingalpha commented May 20, 2022

ashb commented May 20, 2022

zachliu commented May 25, 2022 •

edited

Loading

ashb commented May 27, 2022

maf-rnmourao commented Jun 2, 2022

bbovenzi commented Jun 2, 2022 •

edited

Loading

NilsJPWerner commented Jun 8, 2022 •

edited

Loading

junaidnasir-ps commented Jun 9, 2022

jpipas commented Jun 13, 2022

uranusjr commented Jun 14, 2022

IonutArmeanu-work commented Jun 16, 2022 •

edited

Loading

j-adamczyk commented Jun 22, 2022

kobethuwis commented Sep 21, 2022 •

edited

Loading

cgadam commented Sep 21, 2022 •

edited

Loading

radiophysicist commented Jun 23, 2023

mjkonarski-b commented Jun 26, 2023 •

edited

Loading

potiuk commented Jun 26, 2023

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

Comments

rotemseekingalpha commented May 9, 2022

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

boring-cyborg bot commented May 9, 2022

rotemseekingalpha commented May 9, 2022

jpipas commented May 9, 2022 • edited by uranusjr Loading

bbovenzi commented May 9, 2022 • edited Loading

jpipas commented May 14, 2022

ashb commented May 17, 2022

bbovenzi commented May 18, 2022

rotemseekingalpha commented May 19, 2022 • edited Loading

ashb commented May 19, 2022

jpipas commented May 19, 2022

ashb commented May 19, 2022

rotemseekingalpha commented May 20, 2022

ashb commented May 20, 2022

zachliu commented May 25, 2022 • edited Loading

ashb commented May 27, 2022

maf-rnmourao commented Jun 2, 2022

bbovenzi commented Jun 2, 2022 • edited Loading

NilsJPWerner commented Jun 8, 2022 • edited Loading

junaidnasir-ps commented Jun 9, 2022

jpipas commented Jun 13, 2022

uranusjr commented Jun 14, 2022

IonutArmeanu-work commented Jun 16, 2022 • edited Loading

j-adamczyk commented Jun 22, 2022

kobethuwis commented Sep 21, 2022 • edited Loading

cgadam commented Sep 21, 2022 • edited Loading

radiophysicist commented Jun 23, 2023

mjkonarski-b commented Jun 26, 2023 • edited Loading

potiuk commented Jun 26, 2023

jpipas commented May 9, 2022 •

edited by uranusjr

Loading

bbovenzi commented May 9, 2022 •

edited

Loading

rotemseekingalpha commented May 19, 2022 •

edited

Loading

zachliu commented May 25, 2022 •

edited

Loading

bbovenzi commented Jun 2, 2022 •

edited

Loading

NilsJPWerner commented Jun 8, 2022 •

edited

Loading

IonutArmeanu-work commented Jun 16, 2022 •

edited

Loading

kobethuwis commented Sep 21, 2022 •

edited

Loading

cgadam commented Sep 21, 2022 •

edited

Loading

mjkonarski-b commented Jun 26, 2023 •

edited

Loading