I have prototyped such a template based neural network framework. Based on this, I have implemented a sparse auto-encoder for small size image patches (see data/UFLDL directory for bank of 8x8 patches).

The serial performance is moderate, comparable to a vectorized Matlab implementation. Based on a JVM -Xprof profile, the major cost is calling libBLAS routines.

In current version, the implementation is not fully exposed to breeze. I create one more interface layer for linear algebra (vector and matrix), such that I can possibly change to another numerical library.