Support one-to-many piping in the pipeline syntax

[xiaq]

This issue is filed from #485, which asks for the functionality of piping both the stdout and the stderr of a command to different commands. That bug was closed because the functionality is now possible with the low-level run-parallel and pipe builtins, but no new syntax were introduced.

This issue discusses the possibility of extending the pipeline syntax to support such a pipeline configuration. Citing @mqudsi's comment, it is not easy to come with a unambiguous syntax for this:

The primary issue with with working on both is parsing intent. Presuming stream redirection operators like 1>| and 2>|, how the command

./foo 1>| bar 2>| bar2

is parsed is ambiguous. Is the stderror redirection meant to apply to ./foo or the output of bar? This can only be solved by using brackets:

./foo 1>| bar 2>| bar2
./foo 1>| { bar 2>| bar2 }

A comment about this. I am not sure whether @mqudsi proposes that ./foo 1>| bar 2>| bar2 to mean "pipe stdout of foo to bar, and stderr of foo to bar2", but if that is the case, this is quite counter-intuitive. Traditional pipelines always work in a linear fashion, so it is tempting to interpret this as "pipe stdout of foo to bar, and stderr of bar to bar2".

The syntax for the pipeline should prioritize linear pipelines and make non-linear pipelines more explicit.

---

Traditionally, this functionality is implemented with process substitution:

foo > >(bar) 2> >(bar2)

However, process substitution relies support for either /dev/fd filesystem or named FIFOs. This is backwards: named FIFOs or /dev/fd is indeed needed if the process substitution needs to be used as command arguments, but when used in redirections, the same functionality is entirely implementable with plain, unnamed pipes.

In fact, in Elvish it is already possible to do this, except that you have to manage the lifecycle of pipes manually:

pout = (pipe)
perr = (pipe)
run-parallel {
  foo > $pout 2> $perr
  pwclose $pout
  pwclose $perr
} {
  bar < $pout
  prclose $pout
} {
  bar2 < $perr
  prclose $perr
}

Note that in the first function passed to run-parallel, foo > $pout 2> $perr resembles the process substitution version. This is expected.

---

Now for brainstorming a new syntax!

I think this is a bad idea, but a very intuitive syntax can look like this:

foo | bar
   2| bar2

I have chosen to change 2>| proposed by @mqudsi to 2| for terseness. Like 2>, there must not be any space between 2 and |.

When you have longer pipelines you will need to align them up:

foo | bar | quux
   2| bar2 # applies to stderr of foo
         2| quux2 # applies to stderr of bar

This syntax really takes whitespace-dependent syntax to the extreme. Again I don't think it's a good idea.

Another idea is supporting putting markers on commands in a pipeline, so that they can be referred to later on. Here I use ^name both as marker and reference, but it's likely we will need separate syntax for them:

foo ^f | bar ^b | quux
   ^f 2| bar2
            ^b 2| quux2

The parser can work by looking beyond the pipeline on the first line, and as long as subsequent lines start with a marker, add that to the part of the pipeline.

[zzamboni]

@xiaq First of all, I must say that this is a great feature. I have wished for parallel stdout/stderr pipelines for a long time, and it's amazing that Elvish already supports this. And I also learned about pipe and friends in the process - great work!

About the syntax - honestly I think the following should be clear enough:

cmd | stdout-consumer 2| stderr-consumer

Lamdas are already allowed in pipes, so disambiguation could be done easily, e.g. to my eyes, the following is clear:

cmd | { cmd2 | cmd3 } 2| { cmd4 | cmd4 2| cmd5 }

However, I understand your point about the linearity of pipes. If you want to make it explicity, why not introduce a new builtin (pipesplit comes to mind) that works like this:

pipesplit cmd stdout-consumer stderr-consumer

This could internally map directly into the structure you described with run-parallel.

[zzamboni]

It just dawned on me that this can be implemented as a function, and it works:

fn pipesplit [l1 l2 l3]{
  pout = (pipe)
  perr = (pipe)
  run-parallel {
    $l1 > $pout 2> $perr
    pwclose $pout
    pwclose $perr
  } {
    $l2 < $pout
    prclose $pout
  } {
    $l3 < $perr
    prclose $perr
  }
}

E.g.:

> pipesplit { echo stdout-test; echo stderr-test >&2 } { echo STDOUT: (cat) } { echo STDERR: (cat) }
STDOUT: stdout-test
STDERR: stderr-test

[xiaq]

@zzamboni Right, it is already implementable as a function :)

cmd | stdout-consumer 2| stderr-consumer might seem clear for such simple pipelines, but we need a clear rule for longer pipelines. For example, what about foo | bar 2| lorem 2| ipsum? Whose stderr does ipsum get?

[zzamboni]

@xiaq the rule could be "maximum one | and one 2| per level", and force the user to disambiguate. For example, foo | bar 2| lorem 2| ipsum would give an error because there are 2 stderr-pipeline symbols at the same level. The user would need to specify using lambdas whether he meant foo | bar 2| { lorem 2| ipsum } or foo | { bar 2| lorem } 2| ipsum, both of which would be valid according to the rule.

[zzamboni]

By using lambdas, the user could choose to use indentation to clarify things, as in some of your suggestions above, but the meaning would be clear to Elvish from the nesting, not from the indentation. E.g.:

foo `
| {
  bar 2| lorem
} `
2| ipsum

[zakukai]

I think using lambdas to disambiguate the proposed multiple-pipeline syntax would not actually achieve the desired result. The expectation in a pipe is that it connects a command immediately to the left, to a command immediately to the right. So:

~> A | B 2| C

This really would have to connect B's stderr, not A's, to C. Working around that with lambdas doesn't get us far IMO:

~> { A 2| C } | B      # B gets a combination of stdout and value streams from A and C
~> { A | B } 2| C      # C gets a combination of stderr from A and B

Getting around this just with lambdas in the pipeline would require preventing the undesired output of B or C from reaching the other tool that's being connected to A's other output:

~> { A | B 2> /dev/null } 2| C

...And of course that's not great either: to get B's stderr as part of the whole pipeline's combined stderr, you would need to dup stderr:

~> { { A | B 2>&3 } 2| C } 3>&2

I know we already have a solution in the form of a user-defined function ("pipesplit" shown above) - and it's actually a good solution, but I want to explore the possibilities a bit with regard to syntax:

Consider: the real benefit of redirecting to a process substitution, instead of piping, is that it uses redirection syntax rather than pipeline syntax: Redirections do not chain, instead they stack on the last command in the pipeline, so it's relatively straightforward to attach multiple of them to a single command.

So we could do something like this:

fn pipe-command [cmd]{
  local:p = (pipe)
  { $cmd < $p; prclose $p } &
  put $p
}
~> A stderr>(pipe-command { C }) | B

Personally I like that better, at least expressively speaking, than "split-pipe". It doesn't use /dev/fd (because elvish supports file objects that can be used in redirections), and it's easier to expand truly to "many" pipelines rather than just one or two.

Of course, there are problems with the implementation of "pipe-command" above:

• The write end of the pipe isn't closed until the pipe gets garbage-collected. (Maybe we could have an atomic "redirect to an open file and close it"?)
• The pipe object is retained until "C" terminates. (see above about atomic redirect-and-close)
• "C" can therefore hang: If "C" won't terminate until it gets EOF on its input, that won't occur until the write end of the pipe is closed, which won't happen until the pipe gets GC'ed, which won't happen... until "C" terminates.
• The read end of the pipe is held open until "C" terminates: that means if "C" closes its input, the pipe buffer could fill up and "A" could hang trying to write to stderr (rather than SIGPIPE and probably terminate)
• "C" isn't part of the pipeline job: The job won't wait for "C" to terminate and "C" won't receive suspend or interrupt signals issued to the job's process group, etc.

(Personally I think it would really be preferable to have the two ends of the pipe as separate objects. Relying on GC to clean up a pipe isn't an ideal situation, of course, since it's not immediate - but the two ends are usually handled separately, doesn't really make sense IMO to bind them together.
It should also be noted that some of the issues outlined above apply to "pipesplit" as well: processes in the "pipeline" can't signal each other by closing their individual pipe ends, because the shell holds those pipe ends open until the processes terminate.)

So building on this idea, I think a good answer could be to add syntax that works like "pipe-command" above, except better-managed: The shell doesn't retain the read end of the pipe at all - avoiding the deadlock preventing the pipe from getting GC'ed and the one from A filling up the pipe buffer and hanging if "C" closes its input but doesn't terminate:

~> put (| X)  # "Pipe into X": Run X as a background job, yield a pipe to its input
▶ <pipe{-1 10}>
~> put (Y |)  # "Pipe from Y": Like above but it captures output instead
▶ <pipe{11 -1}>
~> A stderr> (| C) | B   # Connect A's stderr to a pipe into C...

It still doesn't solve the problem of C not being part of the pipeline job, unfortunately. C will run in the background until it self-terminates. But when A terminates, the write end of the pipe to C should be closed (possibly after a GC?), so if C terminates on EOF of its input, it would terminate at that point.

Alternately, if the feature were only allowed as part of a pipeline, "C" could be part of the pipeline job, and the shell would be in a better position to manage the lifetime of the pipe:

~> A stderr>| C | B

Not sure about the syntax (I at least like it better than "A > >(C)"...) but you get the idea: it's a pipe, but with a syntax that behaves like a redirect so that multiple of them would all apply to A rather than chaining.