Plan to develop the inference library of fluid

[Xreki]

Based on my experience in implementing a simple C++ inference example for fluid #7097 , I get a basic understanding of the training and inferring process of fluid.

To implement the inference library for fluid, we need discussion on following things:

• Framework Design
• Refine the Storing Format of Inference Network
• Optimize the Inference Program
• Compiling Aspects

Framework Design

In fluid, the inference process is composed of 2 phases:

• Creation phase. In fluid, two ProgramDescs are needed for inference: one defines the inference network, namely inference_program; the other defines how and where to load the parameters, namely load_program.
  • Basically, proto string of inference network, feed_var_names and fetch_var_names are needed to create the two ProgramDesc.
  • No Tensors or models are created in this phase, inference_program and load_program just hold the protobuf message information of the inference network and initializing the network, respectively.
• Execution phase. In fluid, users can switch among different execution environments flexibility. There are two steps:

  • Configuration step, allowing users to set different configuration, such as different devices (CPU or GPU), different runtime setting (multi-threads or multi-devices), etc.. In fact, this step is used to initialize the execution environment for inference network.

    • Once the configuration step is completed, users can run inference for many-times.
    • Users can initialize several different execution environments for the same inference network.
    • The loading of parameters will be done in this step too, by running the load_program.

  • Running step, allowing users to feed different data, do inference and fetch the predicted results.

    • Supporting the running in both synchronous and asynchronous way.

    

As a result, there should be at least three key concepts in fluid's C++ API:

• InferenceDesc, to hold the handle of inference_program and load_program, which can be initialized from file or from buffer.

• Tensor, an easy-to-use data structure for users. The Tensor and LoDTensor of fluid are too complicated for users. We do not need delayed memory allocation in this structure. Use Tensor and LoDTensor directly in user codes.

• Execution, to hold the configuration of the execution environment.

  [NEED MORE DETAIL]

Refine the Storing Format of Inference Network

Currently, models trained by v2 api cannot be read in C-API directly. We need extra steps:

• Remove the label, cost and evaluator layers in the network manually to get an inference network
• Use dumpy_config.py to get the serialized config, or use merge_model.py to get a merged model (the serialized config and all parameter files are merged into a single file)

In fluid, we hope the model stored in the training process can be used in c++ inference code directly. Currently, there are a couple of interfaces, fluid.io.save_infernece_model and fluid.io.load_inference_model which are specially designed for inference. However, the storing format needs to be refined.

• fluid.io.save_inference_model uses pickle.dump to store program_desc_str, feed_var_names and fetch_var_names information. It is difficult to support pickle in C++ code without third-party tools (like PicklingTools). It's better to design another way to save inference model, like:
  • design a new protobuf data structure for inference
  • or make feed_var_names and fetch_var_names a member of ProgramDesc
  • or store feed_var_names, fetch_var_names and program_desc_str sequentially and use some keywords to separate
• fluid.io.save_inference_model save the serialized protobuf message of the network and all parameter variables into separated files in the same directory. However, many users may hope to save all the parameters into a single file and init the inference model from a buffer. We should enable to merge all the parameter files into one. This may lead to modification of some operators, such as load_op.

Besides, the storing formats of train and inference are different. Users need to save two copies if they want to use the results to fine-tune and infer at the same time. I am not sure whether it is better to unify the storing format.

Optimize the Inference Program

• As shown in #7097, many Variables create, such as velocity_*, learning_rate_* and *@GRAD, but not referenced in any op. There are many unreferenced Variable in the inference program. Prune is called to remove the unreachable operators in the inference program which should remove the unreferenced Variables at the same time.

Compiling Aspects

• All fluid's core C++ binaries should be contained in a single static or shared library, named like libpaddle_fluid.a or libpaddle_fluid.so.
  • libpaddle_fluid.so should link all the dependent libraries but limits the symbol-table at the same time as C-API does.
  • libpaddle_fluid.a should not contain any binaries of third-party libraries (gflags, glog, ...).
• Maybe we need to support the compiler (gcc 4.8.2) that commonly deployed on our developing servers.

[Xreki]

TODO list for inference framework:

1. Framework API (@luotao1, @kavyasrinet, @kexinzhao, @abhinavarora, @sidgoyal78, @Xreki)
   • 1.1) Design Doc (@Xreki Design doc of inference API for fluid #7314 Add design doc of inference API for fluid. #7315 )
   • 1.2) Implementation
2. Storing Format (@kavyasrinet, @kexinzhao, @abhinavarora, @sidgoyal78)
   • 2.1) Get rid of pickle (@kexinzhao Get rid of pickle for Storing format of inference #7221).
     • 2.1.1) Survey of Tensorflow and Caffe2 (@kavyasrinet Survey Tensorflow and Caffe2 to decide standard storing format #7222 )
   • 2.2) Storing to a single file
   • 2.3) Unify the storing format of train and inference
3. Transpiler for inference (@kexinzhao @luotao1, @kavyasrinet, @abhinavarora)
   • 3.1) Optimize the generate inference program
   • 3.2) Write a transpiler
4. Compiling aspects (@luotao1 , @Xreki)
   • 4.1) Build static and shared library (@luotao1 Compiling fluid inference library #7231)
   • 4.2) Build for minimum library size
   • 4.3) Support gcc 4.8.2 in Baidu
5. Tools (@abhinavarora)
   • 5.1) convert v2 model to fluid model
6. Build for mobile devices
   • 6.1) Build for Android
   • 6.2) Build for iOS
7. Speedup for inference

Tasks in bold are discussed in the skype meeting, others will be planed in the future.
Priority: 1.1, 1.2, 2.1 > 2.2, 3.1, 4.1, 5.1

• Framework API:
  • There is no need to design new data structures such as Tensor for users. All the data structures, such as Tensor and LoDTensor, can be directly used in user codes.
  • We still need to design an Inferencer which provides several easy-to-use interfaces, such as init, configuration and run. Configuration is the key interface. It should support all the features of fluid, such as choosing CPU or GPU to do the computation.
• Storing format:
  • need to change save/load_inference_model in https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/fluid/io.py
  • may need to change ProgramDesc in https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto and https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/program_desc.h
  • you can use Add a simple C++ inference example for fluid #7097 to test, which some modification in void InferenceEngine::LoadInferenceModel
  • maybe you need to do some survey (tensorflow, caffe2, etc.) to decide the best storing format
  • if saving all the parameters into a single file, please take care the size of protobuf message (http://www.paddlepaddle.org/docs/develop/documentation/zh/faq/parameter/index_cn.html#a-protocol-message-was-rejected-because-it-was-too-big)
• Transpiler for inference:
  • We can optimize the inference program based on Prune (https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/prune.h#L23) first, then change it to a transpiler.
  • You can use print(inference_program.to_string(True)) in python code to print the proto string to check whether there are unused ops or vars.
• Compiling aspects:
  • You can do your change based on Add a simple C++ inference example for fluid #7097

Once any of you prepare to start one task, please new an issue and let me know.
You are welcome to share your idea if you find other tasks related to inference.