[Doc] Add PyMVA introduction and Keras documentation to TMVA Users Guide

documentation/tmva/UsersGuide/Keras.tex

@@ -0,0 +1,63 @@
+\subsection{Keras}
+\label{sec:keras}
+
+Keras (\url{www.keras.io}) is a high-level wrapper for the machine learning frameworks Theano (\url{www.deeplearning.net/software/theano/}) and TensorFlow (\url{www.tensorflow.org}), which are mainly used to set up deep neural networks.
+
+\subsubsection{Training and Testing Data}
+
+Please note that data defined as testing data in the dataloader is used for validation of the model after each epoch. Such as stated in section \ref{sec:dnn_test_and_evaluation_set}, a proper evaluation of the performance should be done on a completely independent dataset.
+
+\subsubsection{Booking Options}
+
+The Keras method is booked mainly similar to other TMVA methods such as shown in Code Example \ref{example:keras_booking}. The different part is the definition of the neural network model itself. These options are not set in the options string of TMVA, but these are defined in Python and exported to a model file, which is later on loaded by the method. The definition of such a model using the Keras API is explained in detail in section \ref{sec:keras_model_definition}. The settings in the method's option string manage the training process.
+
+The full list of possible settings that can be used in the option string is presented in Table \ref{opt:mva:keras:options}.
+
+\begin{codeexample}
+\begin{tmvacode}
+factory->BookMethod(dataloader, TMVA::Types::kPyKeras, "Keras", <options>);
+\end{tmvacode}
+\caption[.]{\codeexampleCaptionSize Booking of the \textit{PyKeras} method}
+\label{example:keras_booking}
+\end{codeexample}
+
+\begin{option}[h]
+\input optiontables/MVA__Keras.tex
+\caption[.]{\optionCaptionSize
+     Configuration options reference for PyMVA method \textit{PyKeras}.
+}
+\label{opt:mva:keras:options}
+\end{option}
+
+\subsubsection{Model Definition}
+\label{sec:keras_model_definition}
+
+The model of the neural network is defined in Python using the Keras API, stored to a file and then loaded by the Keras method of PyMVA. This section gives an detailed example how this can be done for a classification task. The final reference to build a Keras model is the Keras documentation itself (\url{www.keras.io}). As well, ROOT ships with TMVA tutorials, which include examples for the usage of the PyKeras method for binary classification, multiclass classification and regression.
+
+Code Example \ref{example:keras_model} shows as an example a simple three-layer neural network, which can be applied for any classification task. After running this script, the model file, e.g. \code{model.h5} in this example, has to be set as the option \texttt{FilenameModel} of the method.
+
+\begin{codeexample}
+\begin{tmvacode}
+from keras.models import Sequential
+from keras.layers.core import Dense
+
+# Define model architecture
+model = Sequential()
+model.add(Dense(64, init='glorot_normal', activation='relu',
+        input_dim=num_input_variables))
+model.add(Dense(num_output_classes, init='glorot_uniform',
+        activation='softmax'))
+
+# Set loss function and optimizer algorithm
+model.compile(loss='categorical_crossentropy', optimizer='Adam',
+        metrics=['accuracy',])
+
+# Store model to file
+model.save('model.h5')
+\end{tmvacode}
+\caption[.]{\codeexampleCaptionSize Definition of a classification model in Python using Keras. The placeholders \code{num_input_variables} and \code{num_output_classes} have to be set accordingly to the specific task.}
+\label{example:keras_model}
+\end{codeexample}
+
+\subsubsection{Variable Ranking}
+A variable ranking is not supported.

documentation/tmva/UsersGuide/MLPs.tex

@@ -533,6 +533,7 @@ \subsubsection{Training of Deep Neural Networks}
 did not decrease for a user-specified number of epochs.
 
 \subsubsection{Test and Evaluation Set}
+\label{sec:dnn_test_and_evaluation_set}
 
 The user should note that this implementation uses the error on the TMVA test set
 to check for convergence. Therefore the TMVA test error reported by TMVA may
@@ -556,6 +557,7 @@ \subsubsection{Booking of the DNN Method}
  a user defined name for the method. The fourth argument is the option string
  holding the options for the method. See Sec.~\ref{sec:usingtmva:booking}
  for more information on the booking.}
+\label{example:DNN_booking}
 \end{codeexample}
 
 The \code{Architecture} parameter is used to specify the backend used for the

documentation/tmva/UsersGuide/PyMVAIntro.tex

@@ -0,0 +1,6 @@
+\section{The PyMVA Methods}
+\label{sec:pymvaClassifiers}
+
+PyMVA is the interface for third-party MVA tools based on Python. It is created to make powerful external libraries easily accessible with a direct integration into the TMVA workflow. All PyMVA methods provide the same plug-and-play mechanisms than the TMVA methods described in section \ref{sec:tmvaClassifiers}.
+
+Because the base method of PyMVA is inherited from the TMVA base method, all options from the Option Table \ref{opt:mva::methodbase} apply for PyMVA methods as well.

documentation/tmva/UsersGuide/TMVAUsersGuide.tex

@@ -144,6 +144,10 @@
 \input SVM
 \input BDTs
 \input RuleFit
+% ------------ PyMVA
+\input PyMVAIntro
+\input Keras
+% ------------ combining
 \input Combining
 %
 % ------------ conclusions

documentation/tmva/UsersGuide/optiontables/MVA__Keras.tex

@@ -0,0 +1,11 @@
+\begin{optiontableAuto}
+FilenameModel & \mc{1}{c}{--} & \mc{1}{c}{--} & \mc{1}{c}{--} & Filename of the neural network model defined by Keras\\
+FilenameTrainedModel & \mc{1}{c}{--} & Placed in weights directory of dataloader & \mc{1}{c}{--} & Filename of the model with trained weights. By default, the file is placed in the weights directory of the dataloader, but you can define with this option a custom path to the model file with trained weights.\\
+BatchSize & \mc{1}{c}{--} & 100 & \mc{1}{c}{--} & Batch size used for a single gradient step during training\\
+NumEpochs & \mc{1}{c}{--} & 10 & \mc{1}{c}{--} & Number of training epochs\\
+Verbose & \mc{1}{c}{--} & 1 & \mc{1}{c}{--} & Verbosity during training \\
+ContinueTraining & \mc{1}{c}{--} & false & \mc{1}{c}{--} & Continue training by loading weights from \texttt{FilenameTrainedModel} at beginning of training\\
+SaveBestOnly & \mc{1}{c}{--} & true & \mc{1}{c}{--} & Store only weights with smallest validation loss\\
+TriesEarlyStopping & \mc{1}{c}{--} & -1 & \mc{1}{c}{--} & Number of epochs with no improvement in validation loss after which training will be stopped. The default or a negative number deactivates this option.\\
+LearningRateSchedule & \mc{1}{c}{--} & '' & \mc{1}{c}{--} & Set new learning rate during training at specific epochs, e.g., set new learning rate at epochs 50 and 70 with \code{'50,0.01;70,0.005'}. By default, there is no schedule set.\\
+\end{optiontableAuto}

tmva/pymva/src/MethodPyKeras.cxx

@@ -22,6 +22,8 @@ ClassImp(MethodPyKeras)
 
 MethodPyKeras::MethodPyKeras(const TString &jobName, const TString &methodTitle, DataSetInfo &dsi, const TString &theOption)
    : PyMethodBase(jobName, Types::kPyKeras, methodTitle, dsi, theOption) {
+   fNumEpochs = 10;
+   fBatchSize = 100;
    fVerbose = 1;
    fContinueTraining = false;
    fSaveBestOnly = true;
@@ -32,12 +34,14 @@ MethodPyKeras::MethodPyKeras(const TString &jobName, const TString &methodTitle,
 
 MethodPyKeras::MethodPyKeras(DataSetInfo &theData, const TString &theWeightFile)
     : PyMethodBase(Types::kPyKeras, theData, theWeightFile) {
-    fVerbose = 1;
-    fContinueTraining = false;
-    fSaveBestOnly = true;
-    fTriesEarlyStopping = -1;
-    fLearningRateSchedule = ""; // empty string deactivates learning rate scheduler
-    fFilenameTrainedModel = ""; // empty string sets output model filename to default (in weights/)
+   fNumEpochs = 10;
+   fBatchSize = 100;
+   fVerbose = 1;
+   fContinueTraining = false;
+   fSaveBestOnly = true;
+   fTriesEarlyStopping = -1;
+   fLearningRateSchedule = ""; // empty string deactivates learning rate scheduler
+   fFilenameTrainedModel = ""; // empty string sets output model filename to default (in weights/)
 }
 
 MethodPyKeras::~MethodPyKeras() {
@@ -50,6 +54,8 @@ Bool_t MethodPyKeras::HasAnalysisType(Types::EAnalysisType type, UInt_t numberCl
    return kFALSE;
 }
 
+///////////////////////////////////////////////////////////////////////////////
+
 void MethodPyKeras::DeclareOptions() {
    DeclareOptionRef(fFilenameModel, "FilenameModel", "Filename of the initial Keras model");
    DeclareOptionRef(fFilenameTrainedModel, "FilenameTrainedModel", "Filename of the trained output Keras model");