307 - Modular DAG log levels

[nikki-t]

Purpose

The modular DAG stage in and stage out tasks generate logs that are too verbose.

Pending consideration:

I modified the existing stage in/out CWL to define a log_level input and have hosted those files here: stage_in and stage_out

We may want to request U-DS define the LOG_LEVEL environment variable as an input to their stage-in.cwl task if we want to continue to use their CWL definitions.

Proposed Changes

• Modify stage in/out log levels and set to a default of 20 [INFO] by modifying stage in/out CWL to accept log_level input.
• Provide a new CWL DAG input parameter for the log level that allows the user to specify the level they would like to use for stage in and stage out.
• Modify the modular entrypoint to include a log level (-l) argument and modify stage in/out commands to include a log level.
• Provide a debug CWL DAG input parameter that sets all log levels to debug including the command to run the stage in, process, and stage out CWL tasks. This option sets debug logs via the entrypoint script set -x command and passes the --debug argument to cwltool commands.
• Pass successful features file to XCOM. The size limit is dependent on the database backend (anywhere from 64KB to 2GB) but I think this successful features file size is small.

Issues

• [Enhancement]: Modular DAG Stage Out step too verbose #307 - [BUG] - Modular DAG Stage Out step too verbose

Testing

Ran in unity-venue-dev on unity-nikki-1 deployment.

• See no-debug.log for logs with a stage in/out log level of 20 and no debug logs.
• See debug.logfor logs with a stage in/out log level of 10 and debug logs.

[nikki-t]

@LucaCinquini - I think I covered all the changes that we discussed so I am going to set this PR as ready for review.

Changes made to log levels:

• Display log level information next to log level number in DAG parameter
• Remove debug DAG parameter and modify entrypoints to use log_level to determine debug logs
• Update CWL DAG to match modular CWL DAG; only include two log levels: 10 (debug) and 20 (info) as the stage in/out are not present in the CWL DAG so do not need more than two log levels

Pending items:

• I updated the modular DAG to use the KubernetesPodOperator to match the CWL DAG. Did we move away from using the SPSKubernetesPodOperator?
• I could not figure out how to suppress the docker pull logs. I did not see a flag or environment variable that could be passed to the cwltool or to docker.

[LucaCinquini]

@nikki-t : PR looks good, it's approved. BTW for now we are not using "SpsKubernetesPodOperator" any more because there is no need to override any functionality from "KubernetesPodOperator".