Skip to content

Separate configurations for colorized and json logs in Task SDK / Celery Executor #51082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
amoghrajesh merged 4 commits into from
Jun 3, 2025
Merged

Separate configurations for colorized and json logs in Task SDK / Celery Executor #51082

amoghrajesh merged 4 commits into from
Jun 3, 2025

Conversation

@highlyavailable
Copy link
Contributor

@highlyavailable highlyavailable commented May 26, 2025

Fixes #50963

This PR separates the control of log format (console vs JSON) from log styling (colors and rich traces) in the Task SDK logging configuration. Previously, the single enable_pretty_log boolean controlled both aspects, making it impossible to have console logs without colors and rich traces.

Problem

  • Single enable_pretty_log boolean controlled BOTH format (console vs JSON) AND styling (colors + rich traces)
  • No way to get console logger without colors and rich traces
  • Celery workers showed color escape sequences as raw text in log ingestion solutions like Splunk

Solution

  • Separated enable_pretty_log (format) from enable_colors (styling)
  • enable_pretty_log=True + enable_colors=False = plain text console logger (no colors, no rich traces)
  • Respects existing colored_console_log configuration parameter
  • Celery workers automatically respect the configuration without code changes
  • Change is backwards compatible since:
    • configure_logging() - uses default enable_colors=None → reads config
    • logging_processors(enable_pretty_log=True) - uses default enable_colors=True

New Behavior Matrix:

enable_pretty_log enable_colors Result
True True Colorized console + rich traces + "%Y-%m-%d %H:%M:%S.%f" timestamps
True False Plain text console + no rich traces + "%Y-%m-%d %H:%M:%S.%f" timestamps
False True/False JSON logger + ISO timestamps + dict exception renderer

Testing

All new & existing tests are passing, as well as the pre-commit checks. I wrote some simple unit tests in the test SDK,

  • test_logging_processors_with_colors(): Verifies colored console renderer
  • test_logging_processors_without_colors(): Verifies non-colored console renderer
  • test_logging_processors_json_format(): Verifies JSON renderer unchanged
  • test_configure_logging_respects_colored_console_log_config(): Tests configuration reading
  • test_configure_logging_explicit_enable_colors(): Tests explicit parameter override
  • test_configure_logging_no_airflow_config(): Tests fallback behavior

Since I am new to the project, I also wrote a short testing script (attached as a .txt) and ran it in my local Breeze environment, here were the results:

demo_test.txt

image

@highlyavailable
Tests for new log configuration
1958091
@highlyavailable
Enhance logging configuration by adding 'enable_colors' parameter to …
0397672 
…'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
uranusjr
uranusjr reviewed May 27, 2025
View reviewed changes
uranusjr
uranusjr reviewed May 27, 2025
View reviewed changes
@amoghrajesh amoghrajesh changed the title Fix/50963 Separate configurations for colorized and json logs in Task SDK / Celery Executor Separate configurations for colorized and json logs in Task SDK / Celery Executor May 27, 2025
amoghrajesh
amoghrajesh reviewed May 27, 2025
View reviewed changes
@amoghrajesh
Copy link
Contributor

amoghrajesh commented May 30, 2025

@highlyavailable would you be able to address review comments so we can target this for Airflow 3.0.2 whose RC cut is on Tuesday?

@highlyavailable
Copy link
Contributor Author

highlyavailable commented May 30, 2025

Yes I will get it done this morning

@highlyavailable
Address review comments: use direct imports and match config naming
6a784ad 
- Remove conditional imports, use direct \
- Rename parameter from \ to \ to match existing config
- Remove unrelated formatting options (show_locals) from color logic
- Update tests to reflect the changes
@highlyavailable
Copy link
Contributor Author

highlyavailable commented May 30, 2025

@highlyavailable would you be able to address review comments so we can target this for Airflow 3.0.2 whose RC cut is on Tuesday?

@amoghrajesh I believe my most recent commit addresses all of the feedback in this thread.

@amoghrajesh
Copy link
Contributor

amoghrajesh commented Jun 2, 2025

@uranusjr does the PR address your review comments?

amoghrajesh
amoghrajesh approved these changes Jun 3, 2025
View reviewed changes
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, more like a nit, can be addressed now or in a followup.

@amoghrajesh amoghrajesh added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Jun 3, 2025
@amoghrajesh amoghrajesh added this to the Airflow 3.0.2 milestone Jun 3, 2025
@amoghrajesh amoghrajesh merged commit e25351d into apache:main Jun 3, 2025
71 checks passed
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 3, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

github-actions bot pushed a commit that referenced this pull request Jun 3, 2025
@highlyavailable
[v3-0-test] Separate configurations for colorized and json logs in Ta…
0cd94ae 
…sk SDK / Celery Executor (#51082)

* Enhance logging configuration by adding 'enable_colors' parameter to 'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
(cherry picked from commit e25351d)

Co-authored-by: Peter Bryant <pwbryant1@gmail.com>
@github-actions
Copy link

github-actions bot commented Jun 3, 2025

Backport successfully created: v3-0-test

Status Branch Result
✅ v3-0-test PR Link

github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 3, 2025
@highlyavailable
[v3-0-test] Separate configurations for colorized and json logs in Ta…
e67d6da 
…sk SDK / Celery Executor (apache#51082)

* Enhance logging configuration by adding 'enable_colors' parameter to 'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
(cherry picked from commit e25351d)

Co-authored-by: Peter Bryant <pwbryant1@gmail.com>
kaxil pushed a commit that referenced this pull request Jun 3, 2025
@github-actions @highlyavailable
[v3-0-test] Separate configurations for colorized and json logs in Ta…
2e07d59 
…sk SDK / Celery Executor (#51082) (#51344)

* Enhance logging configuration by adding 'enable_colors' parameter to 'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
(cherry picked from commit e25351d)

Co-authored-by: Peter Bryant <pwbryant1@gmail.com>
kaxil pushed a commit that referenced this pull request Jun 3, 2025
@github-actions @highlyavailable @kaxil
[v3-0-test] Separate configurations for colorized and json logs in Ta…
0794a70 
…sk SDK / Celery Executor (#51082) (#51344)

* Enhance logging configuration by adding 'enable_colors' parameter to 'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
(cherry picked from commit e25351d)

Co-authored-by: Peter Bryant <pwbryant1@gmail.com>
sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
@highlyavailable @sanederchik
Separate configurations for colorized and json logs in Task SDK / Cel…
5c680cd 
…ery Executor (apache#51082)

* Enhance logging configuration by adding 'enable_colors' parameter to 'logging_processors' and 'configure_logging' functions. This allows for customizable console output with or without colors based on user preference or configuration settings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amoghrajesh amoghrajesh amoghrajesh approved these changes

@uranusjr uranusjr uranusjr approved these changes

@phanikumv phanikumv phanikumv approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

Assignees

No one assigned

Labels

area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Milestone

Airflow 3.0.2

Development

Successfully merging this pull request may close these issues.

Separate configurations for colorised and json logs in Task SDK / Celery Executor

5 participants

@highlyavailable @amoghrajesh @uranusjr @phanikumv @vatsrahul1001