Added the SkipConnection layer and constructor

[bhvieira]

I added a DenseBlock constructor, which allows one to train DenseNets (you can train ResNets and MixNets with this as well, only need change the connection, which is concatenation for DenseNets).

Disclaimer: I created the block for a 3D U-Net, so the assumption here is that whatever layer is inside the block, its output has the same spatial dimension (i.e. all array dimensions excluding the channel and minibatch dimensions) as the input, otherwise the connection wouldn't match. I'm not sure this matches the topology of every DenseNet there is out there, but I suppose this is a good starting point.

No tests yet, will add them as the PR evolve.

I'm open to suggestions! :)

[bhvieira]

Oh, there's also a nomenclature issue. When people think of DenseBlocks they are mostly thinking about densely connected convolutional blocks. I suppose we could call this something else, like SkipConnection or ShortcutBlock. Thoughts?

[MikeInnes]

This seems like a good implementation to me but I wonder if it isn't a bit specific. Maybe others can chime in if this seems like it'd be useful to them. Or perhaps there's some precedent in other frameworks?

I like the idea of calling it SkipConnection, and maybe generalising a bit by making the combiner customisable; e.g. it could be a plain + by default and have documentation for passing in concatenation (assuming + is more common).

[bhvieira]

This seems like a good implementation to me but I wonder if it isn't a bit specific. Maybe others can chime in if this seems like it'd be useful to them. Or perhaps there's some precedent in other frameworks?

Skip connections and deep supervision are quite commonplace nowadays, but I get the point. I'm not sure about other frameworks, I'll investigate.

Though Flux does have a bit of domain-specific methods already, skip connections are everywhere in computer vision and segmentation in general.

[r3tex]

It would be nice if this could be written so that new arrays don't have to be allocated on every forward pass. Models used for real-time inference would benefit.

[MikeInnes]

I'm on board with adding this, with the tests and docs you mentioned.

I'd like to remove the special casing of cat, though; probably better to just make that argument explicit.

[bhvieira]

I'd like to remove the special casing of cat, though; probably better to just make that argument explicit.

Oh yeah, you're completely right about that @MikeInnes . When I started with Flux I had just picked Julia, and wasn't that familiar with it at all. I can do it better now. Would it be better to do it with multiple-dispatch (still casing, but improved in my eyes) or a simple keyword-argument defaulting to connection = cat?

[MikeInnes]

I'm thinking just keep it as a positional argument but without a default, so you can do SkipConnection(m, (a, b) -> cat(a, b, 3)) (or define cat3 or something in your script to make it shorter). I think it's better to be explicit here than to have the meaning of SkipConnection be tied to how we define image models (which might change with autobatching, for example).

[bhvieira]

It would be nice if this could be written so that new arrays don't have to be allocated on every forward pass. Models used for real-time inference would benefit.

@r3tex Thanks for the suggestion! My Julia-fu is weak, is the new, streamlined, version enough? I could look into types as well I guess.

Here's proof gradients propagate correctly:

m = Chain(Dense(1, 10), Dense(10,1))
M = SkipConnection(m, (A,B) -> A .+ B)

using Flux.Tracker
using Flux.Tracker: grad

x = randn(1,20)
y = ones(1,20)
loss(x,y) = sum((M(x) - y).^2)

grads = Tracker.gradient(() -> loss(x, y), params(M))
grads[M.layers[1].W]
grads[M.layers[1].b]

[bhvieira]

Also, is the current show method ok? I haven't invested much time on it actually.

[bhvieira]

Added some tests, still superficial, could add more if asked. Is it okay @MikeInnes ?

[MikeInnes]

This is looking nice! Couple small things: remove the Function annotation since not all callables are functions (you might want to use a Flux model here, for example). SkipConnection should also be listed in the layers docs somewhere. Also, would be great to have a news item for this.

[bhvieira]

@MikeInnes I added a line to docs/src/models/layers.md, under ## Other General Purpose Layers, is that what you meant?

[MikeInnes]

Looks perfect, thanks @bhvieira!

bors r+

[bors]

Build succeeded

• ci/gitlab/staging