Biases never updated

[dlidstrom]

The biases are never updated in the backprop algorithm as it currently stands.
Biases can be included in the array of weights which would simplify things a bit (or keep treating separately but need to update them). Then, each input to the network will need to have a 1 as an extra, last, value. First we can update the code like so:

diff --git a/Tinn.c b/Tinn.c
index 74eb573..9a6fe8b 100644
--- a/Tinn.c
+++ b/Tinn.c
@@ -74,7 +74,7 @@ static void fprop(const Tinn t, const float* const in)
         float sum = 0.0f;
         for(int j = 0; j < t.nips; j++)
             sum += in[j] * t.w[i * t.nips + j];
-        t.h[i] = act(sum + t.b[0]);
+        t.h[i] = act(sum);
     }
     // Calculate output layer neuron values.
     for(int i = 0; i < t.nops; i++)
@@ -82,11 +82,11 @@ static void fprop(const Tinn t, const float* const in)
         float sum = 0.0f;
         for(int j = 0; j < t.nhid; j++)
             sum += t.h[j] * t.x[i * t.nhid + j];
-        t.o[i] = act(sum + t.b[1]);
+        t.o[i] = act(sum);
     }
 }
 
-// Randomizes tinn weights and biases.
+// Randomizes tinn weights.
 static void wbrand(const Tinn t)
 {
     for(int i = 0; i < t.nw; i++) t.w[i] = frand() - 0.5f;
@@ -113,8 +113,7 @@ Tinn xtbuild(const int nips, const int nhid, const int nops)
 {
     Tinn t;
     // Tinn only supports one hidden layer so there are two biases.
-    t.nb = 2;
-    t.nw = nhid * (nips + nops);
+    t.nw = nhid * (nips + nops + 2);
     t.w = (float*) calloc(t.nw, sizeof(*t.w));
     t.x = t.w + nhid * nips;
     t.b = (float*) calloc(t.nb, sizeof(*t.b));

Then the training part needs to be adjusted so that each input row contains an extra column with 1.

Example (for xor training - remember, these are the inputs now):

$$ \begin{pmatrix} 0 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 1 \end{pmatrix} $$

The last column will ensure the bias weight is always included in forward propagation. And backprop will update all weights as usual, including the biases.

[m4r35n357]

I am new to this, so bear it in mind! Apologies if I sound like a dummy, but I am learning as I go along . . .

I made your t.tb changes, and got build errors (the call to calloc obviously failed!), so I went further and remove that field from the header file & deleted all references in the c file. All now seems to build & run as before.

Unfortunately I cannot understand what you mean by "the training part", so I cannot make that change yet.

Can you clarify exactly what I need to change? Do you mean just add a 1 at the end of each line in the training data? I have now done that, but I can't detect any change in the output . . .

Cheers!

[dlidstrom]

Hi @m4r35n357 I have implemented my own neural network which handles the biases correctly. I'm not sure my description above is correct, but you can check here for my own solution. I've tried to keep it similar to tinn. Note that correctly updating biases is necessary for keeping the number of neurons to the minimum.

[m4r35n357]

OK cheers for replying! I have just jumped in with both feet, and have several projects in play ATM (genann, tinn, kann) - thanks for adding another one ;)

Anyhow, I will have a look at yours too, and will find a way eventually . . . what I am looking for to start with is a simple demo system for my optimizers here - https://github.com/m4r35n357/VisualOptimizers - I am bored with "test functions"!

[dlidstrom]

Looks cool 😎 Good luck!