Exercise solutions for Chapter 11 - Batch Normalization fix

[slabko]

In Solution 8.4. (Now try adding Batch Normalization ...) you concluded that with Batch Normalization the model performed even worse. However, I found that there is a little bug in DNNClassifier:

self._graph = tf.Graph()
with self._graph.as_default():
    self._build_graph(n_inputs, n_outputs)

# ... removed 5 lines ... #

# extra ops for batch normalization
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)

# Now train the model!
self._session = tf.Session(graph=self._graph)
with self._session.as_default() as sess:
    self._init.run()

Here extra_update_ops are taken outside of the graph, I beleive this is the issue with Batch Normalization. If you move the line extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) into the with self._graph.as_default(): block, it will work pretty close to the original DNNClassifier or even better.

And I reported this bug as an excuse to tell you again how much I love your book. Thank you very much for your incredible work. I'm enjoying every single chapter of it!

[ageron]

Yikes! You are absolutely right, good catch! I'll update the solution, thank you so much.

And also thanks for your very kind words, I'm very touched. It's fantastic to have such enthusiastic and helpful readers! :))

Cheers,
Aurélien

[ageron]

Just fixed this. Indeed, Batch Normalization manages to reach 99.4% accuracy, instead of 99.32%. Not a huge gain, but still nice to have. Thanks again!

[slabko]

I found that such a margin rather depends on the initial state of the randomizer, so yes, Batch Normalization doesn't really give any advantage in this case, but it doesn't spoil anything either.

BTW, in the Tensorflow's documentation, they mentioned that you can add a dependency to the training operation so that it also triggers extra_update_ops:

optimizer = self.optimizer_class(learning_rate=self.learning_rate)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    training_op = optimizer.minimize(loss)

In this case, you don't even need to retrieve extra_update_ops from the graph and explicitly call sess.run(extra_update_ops, feed_dict=feed_dict)

I found it here: https://www.tensorflow.org/api_docs/python/tf/layers/batch_normalization

It might be something they added recently

Thank you! It's my pleasure! :)

[ageron]

Indeed, this does simplify the code. I mention this in the notebook for chapter 11, just before cell [36], but perhaps I should also mention it more prominently in the book (currently, the control_dependencies() function is just mentioned briefly in chapter 12).
Thanks again for your feedback.

[slabko]

Oh, sorry! It was my bad - I did not notice that 😰