Model serialization

[shuttle1987]

Currently there's an issue with the time between model creation and model training. Specifically there's a persephone.Model object that gets created with one API call but then we run into an issue with keeping that in memory until the model training API call because the training doesn't have to occur on the same machine (a processing node) that the model definition occurred on. Currently we can't pickle the whole persephone.Model object and send that file because of an internal issue in tensorflow not supporting pickling. However pickling the model isn't necessary when no weights are trained, we essentially have a tensorflow model and some other information, we can serialize these separately.

[shuttle1987]

Since updating the pickle protocol to the highest (4) there's a different error coming up:

  File "/home/janis/persephone-web-API/pipfileenv/lib/python3.5/site-packages/connexion/decorators/decorator.py", line 44, in wrapper
    response = function(request)
  File "/home/janis/persephone-web-API/pipfileenv/lib/python3.5/site-packages/connexion/decorators/parameter.py", line 207, in wrapper
    return function(**kwargs)
  File "/home/janis/persephone-web-API/persephone_api/api_endpoints/model.py", line 98, in post
    models_storage_path=Path(flask.current_app.config['MODELS_PATH'])
  File "/home/janis/persephone-web-API/persephone_api/api_endpoints/model.py", line 46, in create_RNN_CTC_model
    pickle.dump(model, pickle_file, protocol=4)
TypeError: cannot serialize '_io.TextIOWrapper' object

Look into using something like https://github.com/mgedmin/objgraph to try to find out what's causing the failed pickle.

Failing this it might be worth using something like https://github.com/jsonpickle/jsonpickle for serialization

[shuttle1987]

It looks like using keras and saving to hdf5 would save a whole lot of these nasty problems with saving the models and makes a lot of other things like restoring easier too

[shuttle1987]

In the interim I'm planning on making a single model persist in memory for development purposes so I can get doing more work on the frontend.

[shuttle1987]

Looks like when the upstream PR persephone-tools/persephone#197 gets merged into Persephone this workaround can be removed in favor of say a LRU cache that just stores the models.

[shuttle1987]

While it could be substantially better there is a solution in place for this that loads the tensorflow checkpoint files from disk. So I'm going to close this for now. Feel free to reopen if there's any problems here.