Proposal: Make pullback always take a single input

[oxinabox]

Right now: (What i will call the Sometimes MultiOutput View)
we interpret a primal function returning a tuple as meaning it has multiple outputs,
and thus that the pullback should have multiple inputs.

The alternative view in this proposal (The Always Single Output View),
is that all julia functions only ever have a single output.
Sometimes that output is a tuple (or other iterables), in which case it is iterated on a,b=f(x) type assignement.
And thus all pullbacks should only ever accept a single input.

An implication of this is that if the primal returned a Tuple, the pushforward should return a Composite{Tuple} of derivatives, rather than a Tuple.

---

Sometime Multioutput (current)

Pros:

• Nice to be able talk about multioutput functions having multinput pullbacks. Feels good.
• Easy to write pullback for multiple output functions, as have them all directly

Cons:

• Can lead to very weird things for things that may or may not return tuples.
  • E.g. the identity primal function only has 1 method, but it needs 2 rrules (https://github.com/JuliaDiff/ChainRules.jl/pull/175/files#diff-61fe5ffefb8f5d33c006125d43a7fcfaR129-R144)
  • map(f, ::Tuple) is probably another. And one we really don't want to be splatting

Always Single output (proposed)

Pros

• Nice to stay "Julia only has single output functions"
• Tuples are not special vs other iterables.
• Avoids splatting when the pullback used, which is probably good for performance (depending how the compiler is feeling)

Cons:

• making pushforward return Composite{Tuple}(a,b,c) not (a, b, c) is a bit wordy.
• dispatches become harded to write pullback((a,b,c)::Tuple{A, B, C}) this related to the The Extensibility Problem, for propagator closures #53 and if it even actually exists (if it doesn't this doens't matter).

[willtebbutt]

FWIW I'm very much in favour of the proposed change.

[oxinabox]

Ok, you are in favor, I am in favor, @MikeInnes is in favour.
@shashi , @YingboMa ?

[MikeInnes]

RE the cons, could you not just replace Composite{Tuple} with Tuple? What's the advantage of Composite{Tuple} over Tuple – it seems like Tuple unambiguously represents the same information.

[oxinabox]

RE the cons, could you not just replace Composite{Tuple} with Tuple? What's the advantage of Composite{Tuple} over Tuple – it seems like Tuple unambiguously represents the same information.

Because chainrules need to return valid differentials.
Like numbers or vectors or other things that define zero and +.
And potentially have other linear operators overloaded on them.

We can potentially automatically insert the Composite using the @frule macro, #44.

---

Another example of treating tuple special shows up in ChainRulesTestUtils
https://github.com/JuliaDiff/ChainRulesTestUtils.jl/blob/8bb99ea5aa867e183b460b541a0a618fd4ca9229/src/ChainRulesTestUtils.jl#L166-L172

[oxinabox]

Another reason why pullbacks should have a single input.
Since every pullback must be a linear operator,
we know that 2*pb(x) == pb(2*x)
this holds also for if there are multiple inputs to the pullback / if that input is multiple things.

If i want to write a test of this it is very easy if everything is single variables, especially if they are good differnetial types that also support scaling.
If i need to deal with splatting appropriately everywhere, and having to broadcast sometimes but not others.
It is much more complex

[oxinabox]

Ok I am completely conviced that we must do this.
Because turns out splatting breaks constant folding.
So we do not want Zygote or any other AD system to have to do splatting and loose out on constant folding