Random SqlAlchemy Error

[GFibrizo]

Problem

Sometimes the sqlalchemy.orm.exc.DetachedInstanceError is throwned from the model_buyer_service.
Some logs:

s5', 's6']", 'features_max': '0.0506801187398187', 'features_min': '-0.12948301186034197', 'target_max': '296.0', 'target_min': '39.0'}
cte_data_owner_4            | 2019-11-12 07:30:35,533 INFO - File file_4.csv found
cte_federated_aggregator_1  | 2019-11-12 07:30:37,813 DEBUG - http://172.24.0.9:5000 "POST /trainings/1ee7868c-051e-11ea-b171-0242ac180002/metrics HTTP/1.1" 200 32686
cte_federated_aggregator_1  | 2019-11-12 07:30:37,815 INFO - Response: <Response [200]> 
cte_federated_aggregator_1  | 2019-11-12 07:30:38,900 INFO - url http://cte_model_buyer:9090/models/1ee7868c-051e-11ea-b171-0242ac180002 
cte_federated_aggregator_1  | 2019-11-12 07:30:38,903 DEBUG - Starting new HTTP connection (1): cte_model_buyer:9090
cte_model_buyer_1           | 2019-11-12 07:30:38,905 INFO - Received update from fed. aggr
cte_model_buyer_1           | 2019-11-12 07:30:39,004 INFO - Updating model from fed. aggr. 
cte_model_buyer_1           | 2019-11-12 07:30:39,579 ERROR - Instance <Model at 0x7f5b59bf1ef0> is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
cte_model_buyer_1           | 2019-11-12 07:30:39,580 ERROR - Exception on /models/1ee7868c-051e-11ea-b171-0242ac180002 [PATCH]
cte_model_buyer_1           | Traceback (most recent call last):
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1832, in full_dispatch_request
cte_model_buyer_1           |     rv = self.dispatch_request()
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1818, in dispatch_request
cte_model_buyer_1           |     return self.view_functions[rule.endpoint](**req.view_args)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
cte_model_buyer_1           |     resp = resource(*args, **kwargs)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/flask/views.py", line 88, in view
cte_model_buyer_1           |     return self.dispatch_request(*args, **kwargs)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
cte_model_buyer_1           |     resp = meth(*args, **kwargs)
cte_model_buyer_1           |   File "/app/model_buyer/resources/models_resource.py", line 140, in patch
cte_model_buyer_1           |     return ModelBuyerService().update_model(model_id, data)
cte_model_buyer_1           |   File "/app/model_buyer/services/model_buyer_service.py", line 155, in update_model
cte_model_buyer_1           |     ordered_model, diffs, partial_diffs = self._update_model(model_id, data, BuyerModelStatus.IN_PROGRESS.name)
cte_model_buyer_1           |   File "/app/model_buyer/services/model_buyer_service.py", line 178, in _update_model
cte_model_buyer_1           |     ordered_model.set_weights(weights)
cte_model_buyer_1           |   File "/app/model_buyer/models/model.py", line 104, in set_weights
cte_model_buyer_1           |     self.model.set_weights(weights)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 276, in __get__
cte_model_buyer_1           |     return self.impl.get(instance_state(instance), dict_)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 677, in get
cte_model_buyer_1           |     value = state._load_expired(state, passive)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 660, in _load_expired
cte_model_buyer_1           |     self.manager.deferred_scalar_loader(self, toload)
cte_model_buyer_1           |   File "/root/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 913, in load_scalar_attributes
cte_model_buyer_1           |     "attribute refresh operation cannot proceed" % (state_str(state))
cte_model_buyer_1           | sqlalchemy.orm.exc.DetachedInstanceError: Instance <Model at 0x7f5b59bf1ef0> is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
cte_federated_aggregator_1  | 2019-11-12 07:30:39,588 DEBUG - http://cte_model_buyer:9090 "PATCH /models/1ee7868c-051e-11ea-b171-0242ac180002 HTTP/1.1" 500 177
cte_federated_aggregator_1  | 2019-11-12 07:30:39,589 ERROR - 500 Server Error: INTERNAL SERVER ERROR for url: http://cte_model_buyer:9090/models/1ee7868c-051e-11ea-b171-0242ac180002

To Reproduce

Steps to reproduce the behavior:

1. Having trained at least one model previously, create new model from model buyer ui
2. Accept the training from 4 of the data owners using the script
3. Accept the training from the last data owner (the one in localhost:4004) clicks the button 'Train model' in the UI.
4. See error in logs

Env data

• OS: Linux Mint 19
• Browser: Chrome