Please do not deprecate Dense(...;initb=...)

[vpuri3]

Please do not deprecate initb keyword argument to Dense, as i need to initialize b to nonzero values for many of my models

Flux.jl/src/layers/basic.jl

Line 135 in 9931730

Base.depwarn("keyword initb is deprecated, please simply supply the bias vector, bias=initb(out)", :Dense)

[ToucheSir]

Is there a reason you can't pass in a bias array with the nonzero values you want? Does require doing your own weight initialization as well, but Dense(glorot_uniform(in, out), custom_bias(out), activation) isn't particularly difficult or verbose.

[DhairyaLGandhi]

I think if one is looking to change the behaviour only of a certain param they should be allowed to do so, especially if it's as simple as feeding in a function. The library did support that and keeping initb isn't too hard.

[CarloLucibello]

Dense (and all other layers) can be fed with a bias without also having to pass the weight.
This is clear from the deprecation warning though it's not explained in the docstring:

julia> d = Dense(2, 3, bias=ones(3))
Dense(2, 3)         # 9 parameters

julia> d.bias
3-element Vector{Float64}:
 1.0
 1.0
 1.0

help?> Dense
search: Dense DenseArray DenseVector DenseMatrix DenseVecOrMat DenseConvDims DimensionMismatch codeunits ncodeunits DepthwiseConvDims

  Dense(in, out, σ=identity; bias=true, init=glorot_uniform)
  Dense(W::AbstractMatrix, [bias, σ])

  Create a traditional Dense layer, whose forward pass is given by:

  y = σ.(W * x .+ bias)

  The input x should be a vector of length in, or batch of vectors represented as an in × N matrix, or any array with size(x,1) == in. The out y will be a vector of length out, or a batch with size(y) == (out,
  size(x)[2:end]...)

  Keyword bias=false will switch off trainable bias for the layer. The initialisation of the weight matrix is W = init(out, in), calling the function given to keyword init, with default glorot_uniform (@doc
  Flux.glorot_uniform). The weight matrix and/or the bias vector (of length out) may also be provided explicitly.

  Examples
  ≡≡≡≡≡≡≡≡≡≡
...

[DhairyaLGandhi]

I think it's neat to be able to give a function where the size of the bias is handled per the rest of the information in the constructor.

[DhairyaLGandhi]

Well, the function as an init (Dense(..., init/bias = fn)) does seem easy to retain. It also means there is lesser noise in the api signature.

[darsnack]

We could add that by defining:

create_bias(weights::AbstractArray, bias, dims::Integer...) = bias(dims...)

Then this should work: Dense(in, out; bias = myinit).

[darsnack]

Ah double time

[CarloLucibello]

Actually
" The weight matrix and/or the bias vector (of length out) may also be provided explicitly."
in the docstring may be enough explanation, I think we can leave things as they are.
I'm against adding back a dedicated keyword, there is really no need for it. @darsnack's proposal is doable, although feels a bit we are overcharging the keyword meaning

[vpuri3]

Yes, it is true that there are multiple ways to get the same effect. However, Dense(in,out,act;initb=initFun) is clearer than Dense(in,out,act;bias=initFun(out)).

[ToucheSir]

This becomes a matter of API design philosophy and how much surface area + internal complexity you accommodate. PyTorch, for example, doesn't allow for any customization of init in the constructor and requires one to manually update weights in place after the fact. Keras goes the opposite direction by offering a ton of kwargs for init, but notably doesn't allow for passing in parameter arrays directly.

My understanding is that the rationale for Flux was first articulated in #671 and implemented in #1440. Pretty ambivalent on which approach is better, personally, but I'm also hard-pressed to find a compelling counterexample.

[darsnack]

However, Dense(in,out,act;initb=initFun) is clearer than Dense(in,out,act;bias=initFun(out)).

That's fair, though I must admit that I do not see a big difference in clarity. The main reason for moving away from the initb/initW API is that b/W are not consistent across layers. So, using the initialization API required knowing the internal field names, and even worse was that layers with the same conceptual parameters (weight/bias) did not use the same field names. In the end, it was an API that wasn't easy to use without reading the docs per layer. The current API aims to be consistent across all layers by providing a single init keyword.

I wish we could support every possible stylistic combination, but it quickly turns quite complex to support all of them. Plus, in Flux, we err on the side of not having multiple keyword arguments that all affect the same thing but with some arbitrary precedence of which keyword can override another. To me, the penalty of adding initb as an additional option is greater than the benefits.

[DhairyaLGandhi]

Why was this closed?