Pipe stdout and stderr to two separate commands

[mqudsi]

I haven't dug into this too deeply, but by looking through the documentation and testing a few things, it does not seem that it is possible to operate on both stdout and stderr streams simultaneously (but separately)

~>sh -c "echo stdout; echo 1>&2 stderr"
stdout
stderr

~>sh -c "echo stdout; echo 1>&2 stderr" | (or ?(egrep nostdout) true)
stderr

~> sh -c "echo stdout; echo 1>&2 stderr" | or ?(egrep nostdout) ?()
stderr
▶ $ok

~/rand> sh -c "echo stdout; echo 1>&2 stderr" | or ?(egrep nostdout) () 2>&1 | or ?(egrep nostderr) ()
stderr
▶ ?(fail 'egrep exited with 1')

Is it not possible to redirect both?

[xiaq]

The pipe symbol always connect stdout of the previous command with the stdin of the next command. stderr does not participate in pipelining; I do not know of any shells where stderr does.

[mqudsi]

@xiaq thanks, I supposed as much. Powershell can do it if you use the process api, but that's mainly cheating.

I cannot stress how amazing (and unique) it would be if it were possible to come up with a nice, elegant way of operating on both stderr and stdout separately, without blocking, in elvish. It would be the killer feature everyone needs and they don't even realize it, and elvish would be the only shell to provide that out-of-the-box.

The primary issue with with working on both is parsing intent. Presuming stream redirection operators like 1>| and 2>|, how the command

./foo 1>| bar 2>| bar2

is parsed is ambiguous. Is the stderror redirection meant to apply to ./foo or the output of bar? This can only be solved by using brackets:

./foo 1>| bar 2>| bar2
./foo 1>| { bar 2>| bar2 }

[xiaq]

@mqudsi Thanks for clarifying things. For new shell features, it is often much more tricky to come up with a sensible syntax than a sensible semantics. Welcome to shell design :)

There is a more conservative approach though. Elvish already supports building and manipulating pipes with the pipe, prclose and pwclose commands. Redirecting the input to a pipe will use the read end, while redirecting the output will use the write end.

What Elvish still lacks is a construct for running several things in parallel, and wait for them to finish. The pipeline is actually very close to such a construct: in a | b | c, a, b and c are run in parallel, and the pipeline waits for all of them to finish. However, the pipeline also connects the I/O in a standard manner (stdout of a to stdin of b, and stdout of b to stdin of c), and in the use case under question we want to override this behaviour.

What can work is a (say) builtin run-parallel that takes lambdas, runs them in parallel and wait for them to finish. For instance, run-parallel { a } { b } { c } does the same thing as a | b | c, except that it does not do any redirections. Together with pipe manipulation commands, your example can be implemented as follows:

pout = (pipe)
perr = (pipe)
run-parallel {
  foo > $pout 2> $perr
  pwclose $pout
  pwclose $perr
} {
  bar < $pout
  prclose $pout
} {
  bar2 < $perr
  prclose $perr
}

[krader1961]

FWIW, @xiaq, I don't see where any of popen, pipe, prclose, or pwclose are documented. And popen does not appear to be valid. Having said that I am intrigued by the example you wrote which shows that more basic primitives than a | b can be useful in solving complex shell scripting problem. Especially without having to introduce more arcane syntax (something bash/zsh loves to do).

[xiaq]

@krader1961 Sorry, popen should be pipe, I fixed the original comment.

Good catch for missing doc. I updated the source code but cannot push to prod today, my VPS seems blocked by GFW :-/

[xiaq]

Updated: https://elvish.io/ref/builtin.html#pipe

I also documented fopen and fclose.

[xiaq]

I have added a run-parallel builtin, and the sample code I gave above now works. So the functionality is there, albeit with a somewhat verbose syntax, so I am closing this.

For the possibility of extending the pipeline syntax, please discuss in #500.