Use forward-mode in broadcasting?

[tkf]

It seems that the overhead of the way broadcastings are handled currently is quite large:

• Slow backward pass when the forward pass touches a large array #323 (comment)
• https://discourse.julialang.org/t/zygote-performance/29028/16

Would it make sense to use the ForwardDiff-based solution (broadcast_forward) provided that it can be handled by ForwardDiff? That is to say:

• The function (.f) wrapped in all Broadcasteds are singleton (Base.issingletontype) and
• All arguments (.args) wrapped in all Broadcasteds are Real, AbstractArray{<:Real}, or non-differentiable (e.g., Type, Symbol and Val).

Note that, with ForwardDiff-based approach, we can fuse all broadcastings and also call style-specific materialization method.

Is this a reasonable approach? Or would the performance improvements be implemented soon?

[MikeInnes]

Yes, I think this would be a good approach. I also would like to fill out more explicit gradients for broadcasting.

[cossio]

Does #1001 address this?