Why can the weights of a zero-convolution layer obtain different gradients during training?

[sieve-github-access]

A common understanding in deep learning is that Initializing all the weights with zeros leads the neurons to learn the same features during training [1]. This is because all the weights get exactly the same gradient value and therefore receive exactly the same update during training. This is problematic because the layer effectively have only 1 neuron (aka channel) regardless how many neuron it has.

It seems to me that 1*1 zero-convolution layers, which are used throughout the paper, will suffer from this problem as well,.

That said, the huge success of ControlNet seems to imply that it doesn't suffer from this problem. Could anyone help me understand why?

[geroldmeisinger]

there is a link on the front page with an explanation, see here https://github.com/lllyasviel/ControlNet/blob/main/docs/faq.md

2h video from a guy explaining the paper https://www.youtube.com/watch?v=Mp-HMQcB_M4

[sieve-github-access]

The link doesn't explain why the weights of a zero-convolution layer won't be all the same , which is what this question is about ---
All weights starts with zero, receives the same gradient, so will they all be the same after training ?

[xiechun-tsukuba]

I am here to ask the same question.
the FAQ page only explains why the gradients are not zeros, but the real question is why the gradients are not the same for all the neurons in the zero initialized layer