neural-network

Building a Neural Network from scratch in python with just Numpy and maths. Link to colab notebook: https://colab.research.google.com/drive/1uzdhvDRVAgCURUl2lOY_ISkUv-_bNFiE

a simple two-layer neural network trained on the MNIST digit recognizer dataset.

The NN has a simple two-layer architecture. Input layer $a^{[0]}$ has 784 units corresponding to the 784 pixels in each 28x28 input image. A hidden layer $a^{[1]}$ has 10 units with ReLU activation, and finally the output layer $a^{[2]}$ has 10 units corresponding to the ten digit classes with softmax activation.

Forward propagation

$$Z^{[1]} = W^{[1]} X + b^{[1]}$$

$$A^{[1]} = g_{\text{ReLU}}(Z^{[1]}))$$

$$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$$

$$A^{[2]} = g_{\text{softmax}}(Z^{[2]})$$

Backward propagation

$$dZ^{[2]} = A^{[2]} - Y$$

$$dW^{[2]} = \frac{1}{m} dZ^{[2]} A^{[1]T}$$

$$dB^{[2]} = \frac{1}{m} \Sigma {dZ^{[2]}}$$

$$dZ^{[1]} = W^{[2]T} dZ^{[2]} .* g^{[1]\prime} (z^{[1]})$$

$$dW^{[1]} = \frac{1}{m} dZ^{[1]} A^{[0]T}$$

$$dB^{[1]} = \frac{1}{m} \Sigma {dZ^{[1]}}$$

Parameter updates

$$W^{[2]} := W^{[2]} - \alpha dW^{[2]}$$

$$b^{[2]} := b^{[2]} - \alpha db^{[2]}$$

$$W^{[1]} := W^{[1]} - \alpha dW^{[1]}$$

$$b^{[1]} := b^{[1]} - \alpha db^{[1]}$$

Vars and shapes

Forward prop

• $A^{[0]} = X$: 784 x m
• $Z^{[1]} \sim A^{[1]}$: 10 x m
• $W^{[1]}$: 10 x 784 (as $W^{[1]} A^{[0]} \sim Z^{[1]}$)
• $B^{[1]}$: 10 x 1
• $Z^{[2]} \sim A^{[2]}$: 10 x m
• $W^{[1]}$: 10 x 10 (as $W^{[2]} A^{[1]} \sim Z^{[2]}$)
• $B^{[2]}$: 10 x 1

Backprop

• $dZ^{[2]}$: 10 x m ($~A^{[2]}$)
• $dW^{[2]}$: 10 x 10
• $dB^{[2]}$: 10 x 1
• $dZ^{[1]}$: 10 x m ($~A^{[1]}$)
• $dW^{[1]}$: 10 x 10
• $dB^{[1]}$: 10 x 1