[Enhancement]: Modular DAG Stage Out step too verbose

[mcduffie]

Every file generated by the stage out process does not need to be printed to the log. This is an issue when generating many STAC dataset files with lots of metadata and geolocation. I ended up with a 21 MB log file. It made then browser unresponsive.

[mcduffie]

Example log file:
s3://unity-ops-sps-airflowlogs/dag_id=cwl_dag_modular/run_id=manual__2025-02-05T23:18:09+00:00/task_id=cwl_task_processing/attempt=1.log

[mcduffie]

To further reduce verbosity, set the LOG_LEVEL environment variable to 20

See LOG_LEVEL description in https://unity-sds.gitbook.io/docs/developer-docs/data/docs/users-guide/stage-out

[LucaCinquini]

Possibly coordinate the Airflow LOG level with the STAGE IN/OUT LOG level, and make it a configurable parameter.

[nikki-t]

I think I was able to reduce the verbosity of both the stage in/out logs plus the logs generated from the entrypoint scripts (see PR #315) but it has been harder to make the log levels generated from running the containerized CWL tasks match those generated from the KubernetesPodOperator.

Here is an example of the mismatch:

[2025-02-11, 21:21:40 UTC] {pod_manager.py:471} INFO - [base] 2025-02-11T21:21:40 container_log WARNING This is a 'warning' log
[2025-02-11, 21:21:40 UTC] {pod_manager.py:471} INFO - [base] 2025-02-11T21:21:40 container_log ERROR This is an 'error' log
[2025-02-11, 21:21:40 UTC] {pod_manager.py:471} INFO - [base] 2025-02-11T21:21:40 container_log CRITICAL This is a 'critical' log

Where the KubernetesPodOperator (pod_manager.py) log level is INFO and the logs generated by the CWL docker container are set to WARNING, ERROR, or CRITICAL.

I am not sure if we could modify the KubernetesPodOperator to remove the log level? I haven't been able to find a lot of info on the configuring logs for this operator. We may be able to create a custom class that removes the log level but I don't think we can make it match.

[LucaCinquini]

Nikki did everything that could be done so far, closing the ticket. James's use case is satisfied.