Output is not written to disk if job is marked as stalled by the watchdog

[chrisburr]

For LHCb production jobs the stdout/stderr is continiously written to disk however for user jobs it's only written after the process exits and for stalled jobs it gets lost by the Watchdog object.

Debugging stalled user jobs would be easier if a callback was used to write it continiously.

[fstagni]

Of which output are you talking about exactly? I am a bit confused on why you say that for LHCb production jobs the output is written. There is no output sandbox written out for stalled jobs in any case.

[chrisburr]

Assuming I remember correctly:

• I mean jobs which are marked failed with the Watchdog identified this job as stalled-like message.
• LHCb production jobs continuously write the output here
• LHCb user jobs only write the output at the end of the job here (I'm less sure I've got the right place here)

[fstagni]

Seems like it's the same idea you wrote yesterday in #6559 ...