Neural_Network

About

This is a simple project to implement neural network in c++, the structure of network is inspired by ConvNetJS, Darknet, Caffe and PyTorch.

The main motivation for me to write this framework is the homework from NTHU EE231002 Lab14 Image Processing. There is a task need to add a box around my head and I want it to do automatically, when I discover this paper, MTCNN, I know it is time for me to construct a neural network from scratch.

First, I found this on youtube, 10.1: Introduction to Neural Networks - The Nature of Code, I learn some knowledge about neural network, but javascript? I think for a while, my poor coding ability and poor algorithm may make the whole project unusable or take uncountable time to run, so i turn to c++. (Maybe it is my bias that I think c/c++ is more efficient than javascript) At early stage, I take a reference from ConvNetJS, the code is easy to understand, even until now, the trainer of this project is mostly indentical to that.

After finishing MTCNN, aiming to construct more complex network, I look for some object detection model, but most of them are based on Tensorflow or PyTorch, if I have any problem during implementation, it is hard to trace code to check if I am right or wrong. Then, I found YOLO, it is based on Darknet, a framework that not so complex, human readiable, give me a chance to build more complex network by myself. There are lots of layers in this project based on Darknet but maybe easier to read than the original version.

At the time I finish YOLO, I find that my layer management is too complex due to bad data structure. I noticed Caffe's layer registry by accident, that is what I want. Revise the code right away to manage my layer like that, creating new format to define network structure called otter to store network topology replacing the old tedious method.

SSD...
RNN...
FASTER_RCNN...

Introduce new Tensor data structure based on libtorch, try to implement autograd system with static and dynamic graph. Most of work in./Neural_Network/OTensor/folder are based on PyTorch with little revision.

Feature

• C++11
• No dependencies
• Multi-thread support with OpenMp
• Run only on CPU
• Structure visualization by Netron with Caffe2 like prototxt file
• Easy to add custom layer
• Python interface

Supported Layers

Data layer

• Input layer (raw data input)
• Data layer (support data transform)

Vision layers

• Convolution layer (depthwise support)
• Pooling layer (maxpooling)
• AvgPooling layer
• UpSample layer

Common layers

• Dropout layer
• FullyConnected layer

Activation layers

• Sigmoid layer
• Tanh layer
• Relu layer
• PRelu layer
• LRelu layer
• Mish layer
• Swish layer
• Elu layer

Normalization layer

• BatchNormalization layer

Utility layers

• Concat layer (multi layers)
• Eltwise layer (multi layers)
• ShortCut layer (single layer)
• ScaleChannel layer

Loss layers

• Softmax layer (with cross entropy loss)
• EuclideanLoss layer

Special layers

• Yolov3 layer
• Yolov4 layer

Supported Trainers

• SGD
• ADADELTA
• ADAM

Supported Model

• MTCNN
• YOLOv3
• YOLOv3-tiny
• YOLOv3-openimage
• YOLOv4
• YOLOv4-tiny
• YOLOv4-csp

Construct network

Initialize network

Declear the nerual network. If the backpropagated path is special, you should name it and define the backpropagated path in Neural_Network::Backward() at Neural_Network.cpp.

Neural_Network nn("network_name");    // The default name is sequential

Add layers

It will add layer to neural network, checking the structure of input tensor at some layer, for instance, Concat layer, the input width and height should be the same as all input. Note: The first layer of network should be Input layer or custom Data layer.

nn.addLayer(LayerOption{{"type", "XXX"}, {"option", "YYY"}, {"input_name", "ZZZ"}, {"name", "WWW"}});    // The options are unordered

Data layer

• Input layer options

input_width
input_height
input_dimension

• Data layer options

scale (1)
mean (none) usage: "mean_1, mean_2, ...", same dimension as input

Vision layers

• Convolution layer options

number_kernel
kernel_width
kernel_height ( = kernel_width)
stride (1)
stride_x (-1)
stride_y (-1)
dilation (1)
padding (0)
groups (1)
batchnorm (none)
activation (none)

• Pooling layer (output_size = (input_size + padding - kernel_size) / stride + 1)

kernel_width
kernel_height ( = kernel_width)
stride (1)
padding (0)

• AvgPooling layer
• UpSample layer

stride

Common layers

• Dropout layer

probability (0.5)

• FullyConnected layer options

number_neurons
batchnorm (none)
activation (none)

Activation layers

• Sigmoid layer
• Tanh layer
• Relu layer
• PRelu layer

alpha (0.25)

• LRelu layer

alpha (1)

• Mish layer
• Swish layer
• Elu layer

alpha (0.1)

Normalization layer

• BatchNormalization layer

Utility layers

• Concat layer (multi layers)

concat (none)
splits (1)
split_id (0)

• Eltwise layer

eltwise
eltwise_op (prod, sum, max)

• ShortCut layer (single layer)

shortcut
alpha (1)
beta (1)

• ScaleChannel layer

scalechannel

Loss layers

• Softmax layer
• EuclideanLoss layer

Special layers

• YOLOv3 layer

total_anchor_num
anchor_num
classes
max_boxes
anchor
mask
ignore_iou_threshold (0.5)
truth_iou_threshold (1)

• YOLOv4 layer

total_anchor_num
anchor_num
classes
max_boxes
anchor
mask
ignore_iou_threshold (0.5)
truth_iou_threshold (1)
scale_x_y (1)
iou_normalizer (0.75)
obj_normalizer (1)
cls_normalizer (1)
delta_normalizer (1)
beta_nms (0.6)
objectness_smooth (0)
label_smooth_eps (0)
max_delta (FLT_MAX)
iou_thresh (FLT_MAX)
new_coordinate (false)
focal_loss (false)
iou_loss (IOU)
iou_thresh_kind (IOU)

Add output

The output of network can be more than one, if you need, just type this command. The default output is the last layer of network.

nn.addOutput("Layer_name");

Construct network

It will construct the static computation graph of neural network automatically, but not checking is it reasonable or not.

nn.compile(mini_batch_size);    // The default mini_batch_size is 1

Network shape

Show the breif detail between layer and layer.

nn.shape();    // Show network shape

Visualization

It will output a Caffe2 like network topology file, but it is not a converter to convert the model to Caffe2.

nn.to_prototxt("output_filename.prototxt");    // The default name is model.prootxt

Open the file at Netron to see the network structure.

Example of constructing a network

The example is a classifier with 3 classes, and the input is 28x28x3 tensor, it works may not be so well, just try to demonstrate all kinds of layer.

Neural_Network nn;
nn.addLayer(LayerOption{{"type", "Input"}, {"input_width", "28"}, {"input_height", "28"}, {"input_dimension", "3"}, {"name", "input"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "16"}, {"kernel_width", "3"}, {"stride", "2"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "Relu"}, {"name", "conv_1"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "16"}, {"kernel_width", "3"}, {"stride", "1"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "LRelu"}, {"name", "conv_2"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "8"}, {"kernel_width", "1"}, {"stride", "1"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "LRelu"}, {"name", "conv_3"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "16"}, {"kernel_width", "3"}, {"stride", "1"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "LRelu"}, {"name", "conv_4"}});
nn.addLayer(LayerOption{{"type", "ShortCut"}, {"shortcut", "lr_conv_2"}, {"name", "shortcut_1"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "8"}, {"kernel_width", "1"}, {"stride", "1"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "Mish"}, {"name", "conv_5"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "8"}, {"kernel_width", "1"}, {"stride", "1"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "Mish"}, {"name", "conv_6"}, {"input_name", "re_conv_1"}});
nn.addLayer(LayerOption{{"type", "Concat"}, {"concat", "mi_conv_5"}, {"splits", "1"}, {"split_id", "0"}, {"name", "concat"}});
nn.addLayer(LayerOption{{"type", "Concat"}, {"splits", "2"}, {"split_id", "1"}, {"name", "concat_4"}});
nn.addLayer(LayerOption{{"type", "Dropout"}, {"probability", "0.2"}, {"name", "dropout"}});
nn.addLayer(LayerOption{{"type", "Convolution"}, {"number_kernel", "16"}, {"kernel_width", "3"}, {"stride", "2"}, {"padding", "same"}, {"batchnorm", "true"}, {"activation", "Swish"}, {"name", "conv_7"}, {"input_name", "concat"}});
nn.addLayer(LayerOption{{"type", "Pooling"}, {"kernel_width", "2"}, {"stride", "2"}, {"name", "pool_1"}, {"input_name", "concat"}});
nn.addLayer(LayerOption{{"type", "ShortCut"}, {"shortcut", "sw_conv_7"}, {"name", "shortcut_2"}});
nn.addLayer(LayerOption{{"type", "UpSample"}, {"stride", "2"}, {"name", "upsample"}});
nn.addLayer(LayerOption{{"type", "Concat"}, {"concat", "dropout"}, {"splits", "1"}, {"split_id", "0"}, {"name", "concat_3"}});
nn.addLayer(LayerOption{{"type", "AvgPooling"}, {"name", "avg_pooling"}, {"input_name", "concat"}});
nn.addLayer(LayerOption{{"type", "ShortCut"}, {"shortcut", "avg_pooling"}, {"name", "shortcut_3"}, {"input_name", "concat_3"}});
nn.addLayer(LayerOption{{"type", "FullyConnected"}, {"number_neurons", "32"}, {"name", "connected"}, {"activation", "PRelu"}});
nn.addLayer(LayerOption{{"type", "FullyConnected"}, {"number_neurons", "3"}, {"name", "connected_2"}});
nn.addLayer(LayerOption{{"type", "Softmax"}, {"name", "softmax"}});
nn.compile();
nn.to_prototxt();

The graph shown by Nerton.

Forward Propagation

The data flow of network is based on Tensor. To forward propagation, just past the pointer of data to network.Forward(POINTER_OF_DATA) function. And it will return a pointer to Tensor pointer. Careful to use the output, it is the direct result of Neural Network!

Tensor data(1, 3, 28, 28);
Tensor** output = nn.Forward(&data);

Backward Propagation

To backward propagation, just past the pointer of data to network.Backward(POINTER_OF_LABEL) function. And it will return the loss with floating point type.

Tensor label(1, 1, 1, 3); label = {0, 1, 0};
float loss = nn.Backward(&label);

Extract Temporary Result

If you want to extract some result from the inner layer,

Tensor temp;
nn.extract("LAYERNAME", temp);

Add custom layer

You can add the custom layer like Caffe. Save model as otter model like below! If defined correctly, it will save everything automatically.

#include "Layer.hpp"
class CustomLayer : public BaseLayer {
public:
    CustomLayer(Layeroption opt);
    void Forward(bool train);    // For normal layer
    void Forward(Tensor *input);    // For input layer
    void Backward(Tensor *target);    // The output tensor should be extended! Or it will cause segmentation fault when clearing gradient (maybe fix at next version)
    vtensorptr connectGraph(vtensorptr input_tensor_, float *workspace);    // If there are multi inputs or need workspace
    inline const char* Type() const {return "Custom_Name";}
private:
    ...
};
REGISTER_LAYER_CLASS(Custom);

Remeber to add enum at Layer.hpp.

In the constrctor of custom layer, you can use ask space for storing data, for example

CustomLayer::CustomLayer(Layeroption opt) : BaseLayer(opt) {
    this->applyInput(NUM);    // ask for input space to store input tensor (default = 1) Note: Data layer should set it to 0
    this->applyOutput(NUM);    // ask for output space to store output tensor (default = 1)
    this->applyKernel(NUM);    // ask for kernel space to store data
    kernel[0] = Tensor(BATCH, CHANNEL, HEIGHTWIDTH, PARAMETER);
    kernel[0].extend();    // If the kernel parameter can be updated
    kernel[1] = ...
    this->applyBias(NUM);    // ask for biases space to store data
    biases[0] = Tensor(BATCH, CHANNEL, HEIGHTWIDTH, PARAMETER);
    biases[0].extend();
    biases[1] = ...
}

If the layer can be trained, you need to pass train arguments to trainer, you need to add code at BaseLayer::getTrainArgs(), return the traing arguments, the traing arguments is defined by,

struct Train_Args {
    bool valid;
    Tensor *kernel;
    Tensor *biases;
    int kernel_size;
    int kernel_list_size;
    int biases_size;
    vfloat ln_decay_list;
};

Get training arguments

If you want to train with your own method, you can use this command to get the whole weights and delta weights in network, if the layer is trainable. It will return a vector of Train_Args.

vector<Train_Args> args_list = network.getTrainArgs();

You can update the weight by your own method, or just use the Trainer below to update the network weight automatically.

Save otter model ♥

If you add some custom layer, remember to write the definition of layer prarmeter in layer.txt, the syntax is like below.

Customed {
    REQUIRED TYPE PARAMETER_NAME // for required parameter (three parameters with two spaces)
    OPTION TYPE PARAMETER_NAME DEFAULT_VALUE // for optional parameter (four parameters with three spaces)
    OPTION multi/single connect PARAMETER    // If layer need extra input
    REQUIRED int net    // If layer need network input size
}

Then, you can save the model without revise any code. The otter file is easy to read and revise but it is sensitive to syntax, edit it carefully. The otter model syntax is like below.

name: "model_name"
output: OUTPUT_LAYER_1_NAME    // optional, can more than one
output: OUTPUT_LAYER_2_NAME
# you can write comment in one line after hash mark
LayerType {
    name: LAYER_NAME    // optional
    input_name: INPUT_LAYER_NAME    // optional
    Param {
        LAYER_PARAMETER: PARAMETER    // Look up the above layer option
        LAYER_PARAMETER: "PARAMETER"    // If the parameter contain space, remember to add quotation mark
    }
    batchnorm: BOOL    // optional
    activation: ACTIVATION_LAYER    //optional
}
LayerType {
    ...
}
...

Just type this command, it will generate one or two file mode_name.otter, model_name.dam, first is the network structure file, second is the network weights file.

nn.save_otter("model_name.otter", BOOL);    // true for saving .dam file

Or you can just save the network weights, by typing this command,

nn.save_dam("weights_name.dam");

New!! Save network structure and weights into one file!!

nn.save_ottermodel("model_name.ottermodel");

Load otter model

You can load the model with different way, structure only.otterfile, weights only.damfile, or structure with weights.ottermodelfile.

Neural_Network nn;
nn.load_otter("model_name.otter", BATCH_SIZE);    

Or just load the weights, by typing this command,

nn.load_dam("weight_name.dam");

New!! Load network structure and weights from one file!!

nn.load_ottermodel("model_name.ottermodel", BATCH_SIZE);

Construct trainer

Initialze the trainer

Trainer trainer(&network, TrainerOption{{"method", XXX}, {"trainer_option", YYY}, {"policy", ZZZ}, {"learning_rate", WWW}, {"sub_division", TTT});

Trainer options

• Trainer::Method::SGD

momentum (0.9)

• Trainer::Method::ADADELTA

ro (0.95)
eps (1e-6)

• Trainer::Method::ADAM

ro (0.95)
eps (1e-6)
beta_1 (0.9)
beta_2 (0.999)

Learning rate policy

• CONSTANT
• STEP

step
scale

• STEPS

steps
step_X
scale_X

• EXP

gamma (1)

• POLY

power (4)

• RANDOM

power(4)

• SIG

gamma (1)

Warmup

• warmup

Start training

There are two method for training.

• Method 1 Put all data and label into two vector of Tensor, and past to the function trainer.train_batch(DATA_SET, LABEL_SET, EPOCH), it will do everything automatically, such as shuffle the data, batch composition, etc. The vtensor is define by std::vector<Tensor>.

vtensor dataset;    // Add data with any way
vtensor labelset;    // Add label with any way
trainer.train_batch(dataset, labelset, EPOCH);

• Method 2 Train the network with single data. Careful to the data and label, it should be extended with your mini_batch_size of network.

Tensor data;
Tensor label;
trainer.train_batch(data, label);

Tensor

It is the data structure used in this neural network, data arrangement is NCHW. The Tensor class is defined by,

class Tensor {
    int batch;
    int channel;
    int height;
    int size;
    float* weight;
    float* delta_weight;
};

Initialize the tensor

• Method 1
  You will get a Tensor t with random value inside.

Tensor t(BATCH, CHANNEL, HEIGHT, WIDTH);

• Method 2
  You will get a Tensor t with identical value PARAMETER inside.

Tensor t(BATCH, CHANNEL, HEIGHT, WIDTH, PARAMETER);

• Method 3
  You will get a Tensor t with the same value as the vector you past in with batch, height and width are equal to 1.

vfloat v{1, 2, 3};
Tensor t(v);

• Method 4
  You will get a Tensor t with the same value as the array you past in.

float f[3] = {1, 2, 3};
Tensor t(f, 1, 1, 3);    // t = [1, 2, 3]

Tensor extend

Allocate the memory of delta_weight in Tensor. To save memory, it will not be allocated in default.

Tensor t(BATCH, CHANNEL, HEIGHT, WIDTH);
t.extend();

Tensor reshape

Reshape the Tensor and clear all data.

Tensor t(BATCH, CHANNEL, HEIGHT, WIDTH);
t.reshape(BATCH, CHANNEL, HEIGHT, WIDTH, EXTEND);

Operation on Tensor

• = (Tensor)
  Deep copy from a to b, including extend.

Tensor a(1, 1, 1, 2, 1);    // a = [1, 1]
Tensor b = a;   // b = [1, 1]

• = (float)
  Set all value as input

Tensor a(1, 1, 1, 3, 0);    // a = [0, 0, 0]
a = 1;  // a = [1, 1, 1]

• = (initializer list)
  Set the previous elements as initialzer list

Tensor a(1, 1, 1, 5, 3);    // a = [3, 3, 3, 3, 3]
a = {1, 2, 4};  // a = [1, 2, 4, 3, 3]

• [INDEX]
  Revise or take the value at INDEX.

Tensor a(1, 1, 1, 5, 3);    // a = [3, 3, 3, 3, 3]
a[2] = 0;   // a = [3, 3, 0, 3, 3]
float value = a[4]; // value = 3

• += (Tensor)

Tensor a(1, 1, 1, 2, 1);    // a = [1, 1]
Tensor b(1, 1, 1, 2, 2);    // b = [2, 2]
a += b; // a = [3, 3] b = [2, 2]

• -= (Tensor)

Tensor a(1, 1, 1, 2, 1);    // a = [1, 1]
Tensor b(1, 1, 1, 2, 2);    // b = [2, 2]
a -= b; // a = [-1, -1] b = [2, 2]

• + (Tensor)

Tensor a(1, 1, 1, 2, 1);    // a = [1, 1]
Tensor b(1, 1, 1, 2, 2);    // b = [2, 2]
Tensor c = a + b;   // c = [3, 3]

• - (Tensor)

Tensor a(1, 1, 1, 2, 1);   // a = [1, 1]
Tensor b(1, 1, 1, 2, 2);   // b = [2, 2]
Tensor c = a - b;   // c = [-1,-1]

• <<
  Print all weights in Tensor.

Python Interface

Neural_Network

Only inference mode now! The neural network is not completed with python interface, just for convenience used with some visualize UI, like matplotlib, etc. Before you use the python interface, you should build the library otter.so first!

Initialize network

Initialize network,

nn = Neural_Network(NETWORK_NAME)

Load ottermodel

Load ottermodel from file,

nn.load_ottermodel(MODEL_NAME)

Network shape

Show the breif detail between layer and layer.

nn.shape()

Forward propagation

The data flow of network is based on Tensor. To forward propagation, just past the data to network.Forward(DATA) function. And it will return a list of Tensor.

data = Tensor(1, 3, 28, 28, 0)
result = nn.Forward(data)

Tensor

Tensor in python version is also not completed yet. Just work with basic operation.

Initialize the tensor

• Method 1
  You will get a Tensor t with identical value PARAMETER inside.

t =  Tensor(BATCH, CHANNEL, HEIGHT, WIDTH, PARAMETER);

Load numpy into tensor

WIth any shape of tensor, it will reshape automatically as numpy.ndarray

t = Tensor()
arr = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
t.load_array(arr)    # Tensor shape (1, 2, 2, 2) with value [1, 2, 3 ,4, 5, 6, 7, 8]
print(t)    # Use print to print out the Tensor

Convert Tensor to numpy

If want to do some data analysis, you can convert the Tensor to numpy

t = Tensor(1, 2, 2, 2, 1)
arr = t.to_numpy()    # [[[1, 1], [1, 1]], [[1, 1], [1, 1]]]

Max value and its index

Get the max value and its index inside Tensor

t = Tensor()
arr = np.array([1, 3, 2])
t.load_array(arr)    # Tensor shape (1, 1, 1, 3) with value [1, 3, 2]
index, value = t.max_index()    # value = 3, index = 1

Build and run

Linux, MacOS

Just do make in the ./Neural_Netowrk/directory. Before make, you can set such options in the Makefile:

• EXEC=EXECUTION_FILE_NAME You can change the execution file name by yourself.
• OPENMP=1 to build with OpenMP support to accelerate Network by using multi-core CPU
• LIBSO=1 to build a library otter.so

If your project need to train the YOLOv4 layer, you should revise OPTS = -Ofast as OPTS = -O2 in Makefile

Windows

If you need to train YOLOv4 layer, you can build with

• $ g++ -Ofast -fopenmp -o nn *.cpp

Else, the isnan() function is not working with -Ofast flag

• $ g++ -O2 -fopenmp -o nn *.cpp

Run

• $ ./nn

Example

The XOR problem example:

#include <iostream>

#include "Neural_Network.hpp"

using namespace std;

int main(int argc, const char * argv[]) {
    Neural_Network nn;
    nn.addLayer(LayerOption{{"type", "Input"}, {"input_width", "1"}, {"input_height", "1"}, {"input_dimension", "2"}, {"name", "data"}});
    nn.addLayer(LayerOption{{"type", "FullyConnected"}, {"number_neurons", "4"}, {"activation", "Relu"}});
    nn.addLayer(LayerOption{{"type", "FullyConnected"}, {"number_neurons", "2"}, {"activation", "Softmax"}});
    nn.compile();

    Tensor a(1, 1, 1, 2, 0);
    Tensor b(1, 1, 1, 2, 1);
    Tensor c(1, 1, 1, 2); c = {0, 1};
    Tensor d(1, 1, 1, 2); d = {1, 0};
    vtensor data{a, b, c, d};
    Tensor a_l(1, 1, 1, 1, 0);
    Tensor b_l(1, 1, 1, 1, 0);
    Tensor c_l(1, 1, 1, 1, 1);
    Tensor d_l(1, 1, 1, 1, 1);
    vtensor label{a_l, b_l, c_l, d_l};

    Trainer trainer(&nn, TrainerOption{{"method", Trainer::Method::SGD}, {"learning_rate", 0.1}, {"warmup", 5}});
    trainer.train_batch(data, label, 100);
    
    printf("Input (0, 0) -> %.0f\n", nn.predict(&a)[0]);
    printf("Input (0, 1) -> %.0f\n", nn.predict(&c)[0]);
    printf("Input (1, 0) -> %.0f\n", nn.predict(&d)[0]);
    printf("Input (1, 1) -> %.0f\n", nn.predict(&b)[0]);
    
    return 0;
}

Doodle Classifier with Python Interface

The simple example to implement a doodle classifier with Google Quick Draw dataset. You should download three category of dataset, cat, fish, and bee and train a model for it using above method.

import otter
import cv2
import matplotlib.pyplot as plt

label_dict={0:"bee",1:"cat",2:"fish"}

def plot_images_labels_prediction(images, labels, prediction, idx, num = 10):
   fig = plt.gcf()
   fig.set_size_inches(12, 14)
   if num > 25: num=25
   for i in range(0, num):
       ax=plt.subplot(5, 5, i+1)
       ax.imshow(images[i], cmap= 'binary')
       title=str(idx[i]) + "." + label_dict[idx[i]] + " (" + "{0:0.2f}".format(prediction[i]) + ")"
       ax.set_title(title, fontsize=10)
       ax.set_xticks([]);
       ax.set_yticks([]);
   plt.show()

if __name__ == "__main__":
    nn = Neural_Network()
    nn.load_ottermodel('doodle_classifier.ottermodel')
    nn.shape()

    images = []
    labels = []
    prediction = []
    idx = []

    cat = np.load('cat.npy')
    fish = np.load('fish.npy')
    bee = np.load('bee.npy')

    data = np.concatenate((cat, fish, bee))
    np.random.shuffle(data)

    for i in range(25):
        a = data[i].reshape((28, 28))

        c = np.array([a, a, a])
        test = Tensor()
        test.load_array(c)

        result = nn.Forward(test)
        index, prob = result[0].max_index()
        images.append(a)
        idx.append(index)
        prediction.append(prob)

    plot_images_labels_prediction(images, labels, prediction, idx, 25)