Bug: Garden exec within a workflow suppresses logs

[jamesylgan]

Bug

Garden exec within a workflow suppresses logs

Current Behavior

Screenshots N/A

I have a script running through a garden exec in a command step (have also tested this with a script step) in a Garden workflow. The step's {steps.step.log} just outputs "Step command includes global options that will be ignored...". Nothing else in the steps object has any of the logs I'd expect from my script. None of my expected logs come out through stdout.

Expected behavior

Logs from click.echo() commands in my script to appear in stdout.

Reproducible example

workflows.garden.yaml

kind: Workflow
steps:
  - name: hideslogs
    script: |
      garden exec --interactive=false whereIWantScriptToExectute "runscript.sh"

runscript.sh
(Python script using @click .echo to stdout)

Whether I define it as a script or a command step, stdout ranges from (1) graph resolution logs then nothing from my actual script to (2) no logs at all.

When I pipe the stdout, same thing.

I could change the script to log to file, but then I'd need to do some file mounting to get that log file out, and it feels like there should be a way to just bubble stdout up through the garden exec

Workaround

Note: Running the garden exec command from the command line seemed to show the logs in stdout that we expected, so the cause of this issue is with the Garden workflow and how it reacts with garden exec, not just garden exec command (when using garden exec --interactive=false whereIWantScriptToExectute logs.log<(runscript.sh))

1. Running a kubectl exec in the script step of the workflow instead of garden exec results in seeing the logs.
2. Running the same garden exec in terminal instead of a workflow results in seeing the logs.

Suggested solution(s)

I'm not sure, honestly. If I were to start debugging this I would:

1. Recreate
2. Use whatever debugging flows exist to see what's happening in garden exec run in a workflow when logs don't appear.

The main issue here is that I'm not sure how to debug Garden, so I'm passing the issue along :)

Additional context

I managed to hit a char* parsing exception when trying to make a workaround, so that might be related. Log dump attached: (when trying garden exec --interactive=false whereIWantScriptToExectute logs.log<(runscript.sh))
error (1).log

Your environment

• OS: Ubuntu 18.04.6 LTS
• How I'm running Kubernetes: On EC2 instance
  garden version == 0.12.48

[stefreak]

@jamesylgan Thank you for reporting this, we'll look into this.

It would be very helpful if you can try some more options and tell us about the result.

Does this also happen with the following workflow, running the command directly from the workflow without a shell in between?

kind: Workflow
steps:
  - name: hideslogs
     command: [exec, "--interactive=false", "whereIWantScriptToExectute", "runscript.sh"]

Did you try with --interactive=true as well?

[jamesylgan]

I did try with --interactive=true as well. Same result.

Does this also happen with the following workflow, running the command directly from the workflow without a shell in between?

I'm not entirely sure what you mean by this one - but we did try running it like in your codeblock, with the same result.