Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable suppress_logs_and_warning in cli when debugging #13180

Merged
merged 3 commits into from
Dec 23, 2020

Conversation

turbaszek
Copy link
Member

In some cases commands like 'dags list' can be used for debug purposes.
The problem is that we are suppresing logs and warnings in some cases
to make the output nice and clean. This commit disable this functionality
if logging level is set to DEBUG.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

In some cases commands like 'dags list' can be used for debug purposes.
The problem is that we are suppresing logs and warnings in some cases
to make the output nice and clean. This commit disable this functionality
if logging level is set to DEBUG.
@turbaszek turbaszek requested review from XD-DENG and mik-laj December 19, 2020 05:00
Copy link
Member

@XD-DENG XD-DENG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself looks good to me.

Meanwhile a quick idea for your consideration (may be incorrect):

I remember the main purpose of introducing suppress_logs_and_warning is that otherwise outputs in YAML/JSON formats cannot be parsed properly in a pipe (via jq for example).

So an alternative solution can be disable suppress_logs_and_warning for table output?

@turbaszek
Copy link
Member Author

So an alternative solution can be disable suppress_logs_and_warning for table output?

I suppose you meant the json and yaml output formats. Sure, this is an option but in my opinion commands should provide requested information and nothing more. Especially that in some cases the errors and warnings (for example from dags) can be a real mess

@XD-DENG
Copy link
Member

XD-DENG commented Dec 21, 2020

Make sense to me, and my opinion about that alternative solution was not strong anyway. Will approve, but please re-push to re-run the CI before we merge this please.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Dec 21, 2020
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@mik-laj
Copy link
Member

mik-laj commented Dec 21, 2020

In my opinion, it's worth adding the --verbose switch to enable logs.

Mainly to make it easier to read log import, which is now only available when you run:

python -c "from airflow.models.dagbag import DagBag; DagBag()"

These logs now don't show up when you start airflow dags liist, so a little switch to bring those logs back would be helpful.

airflow dags list -v

For now, I always remove this decorator for development:
Screenshot 2020-12-21 at 23 48 05

@turbaszek @XD-DENG WDYT?

Copy link
Member

@mik-laj mik-laj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that logs should not be available for other logging levels as well.

@github-actions github-actions bot removed the full tests needed We need to run full set of tests for this PR to merge label Dec 21, 2020
@turbaszek turbaszek requested review from mik-laj and XD-DENG December 22, 2020 16:05
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Dec 23, 2020
@potiuk
Copy link
Member

potiuk commented Dec 23, 2020

Some CLI tests are failing :)

@turbaszek turbaszek requested a review from potiuk December 23, 2020 13:15
@turbaszek
Copy link
Member Author

Some CLI tests are failing :)

Should be fixed now

@potiuk
Copy link
Member

potiuk commented Dec 23, 2020

Random test failing! Merging!

@potiuk potiuk merged commit 40d2d6b into apache:master Dec 23, 2020
@potiuk potiuk deleted the allow-warnings-when-debug branch December 23, 2020 13:37
@kaxil kaxil added this to the Airflow 2.1 milestone Jan 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants