Skip to content

Add support for multi model caching & live reloading #1428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 290 commits into from
Nov 11, 2020
Merged

Add support for multi model caching & live reloading #1428

merged 290 commits into from
Nov 11, 2020

Conversation

RobertLucian
Copy link
Member

@RobertLucian RobertLucian commented Oct 8, 2020 •
edited
Loading

Closes #619, closes #1252.

For the Python, ONNX and Tensorflow predictors, the following functionalities are added:

  • Live model reloading - loads/reloads/unloads models from S3 buckets as the API is running.
  • Multi-model caching - gives the API access to thousands of models while only having a couple loaded in memory.

checklist:

  • run make test and make lint
  • test individual high-level functionalities
    • Python predictor + live reloading of models.
    • Python predictor + caching of models.
    • ONNX predictor + live reloading of models.
    • ONNX predictor + caching of models.
    • TensorFlow predictor + live reloading of models.
    • TensorFlow predictor + caching of models.
  • add more pertinent logs
  • retrieve metadata for the models
  • create examples for each of the above tested functionalities
  • re-test each functionality using the provided examples
  • add documentation
  • create tickets for the remaining tasks

@RobertLucian
Add some boilerplate to clear the head
243d27d
@RobertLucian
Add even more boilerplate
d1181af
@RobertLucian
WIP crons
fd9ec82
@RobertLucian
WIP for model validation on the serving container
8a72779
@RobertLucian
Few comments on lib.model.validation
31d1640
@RobertLucian
Add json converter for model template
6511653
@RobertLucian
WIP add validation for model template
c53909c
@RobertLucian
Add checks/exceptions for model template
4de6270
@RobertLucian
Fix GenericPlaceholder validation
3014205
@RobertLucian
Add recursive model validation and add fixes
3ebf70a
@RobertLucian
Small fix for validate_integer_placeholder
b78fede
@RobertLucian
Replace ExclAlternativePlaceholder w/ OneOfAllPlaceholder
b5ad9d3
@RobertLucian
Implement OneOfAll Placeholder for validation
b6098ed
@RobertLucian
Fix AnyPlaceholder when the folder is empty
15a7d97
@RobertLucian
Add validation for PlaceholderGroup
66730c7
@RobertLucian
Small mods to validation
c1ac84f
@RobertLucian
Add docstrings + add TensorFlowNeuronPredictor
6adb9cc
@RobertLucian
Add validation for models:dir in Python
b690ae6
@RobertLucian
Make certain functions "private"
69b37bd
@RobertLucian
Implement part of the SimpleModelMonitor
9d6ecc4
@RobertLucian
Add LockedFile class and more
580b946
@RobertLucian
WIP SimpleModelMonitor
40f557f
@RobertLucian
Add mechanism for live-model-reloading
3532dd4
@RobertLucian
Few forgotten docstrings
b864f15
@RobertLucian
Add structure to hold models in memory
cd34eca
@RobertLucian
WIP - live-reloading w/ parts of model caching
ef17e85
@RobertLucian
Finalize ONNX client for non-models:dir paths
8e2fc2a
@RobertLucian
Add logic for making predictions when models:dir is set for ONNX client
0d00be4
@RobertLucian
Don't encode when writing to locked file
718b791
@RobertLucian
Renaming types & comment
d5f5e1b
RobertLucian and others added 17 commits November 10, 2020 02:40
@RobertLucian
Prevent script.py from exiting when a cron runs; prevent HandleReload…
8cf7a65 
…ConfigRequest from throwing error
@RobertLucian
Move while loop after the touch of init_script_run.txt
d05ac78
@RobertLucian
Layout fixes for the model validation in the CLI
85ada50
@RobertLucian
Make lint
c5b2ab9
@RobertLucian
Fix bug for the MMC Python/ONNX when dir field is used
0559d54
@RobertLucian
Allow additional files for ONNX/TensorFlowNeuron models
035ffb1
@RobertLucian
Fix merge conflicts from 'master' into feature/multi-model-caching
6c70e45 
# Conflicts:
#	pkg/workloads/cortex/lib/type/predictor.py
@RobertLucian
Merge branch 'master' into feature/multi-model-caching
8dd7fcb
@RobertLucian
Fix error formatting + disallow the addition of extra files for ONNX …
1cb6f73 
…models
@RobertLucian
Merge branch 'master' into feature/multi-model-caching
b63a621
@deliahu
Misc updates
7658d2d
@deliahu
Update models.md
7780e64
@RobertLucian
Fix TensorFlow + Inferentia + 1 process not live reloading
bee8d0a
@RobertLucian
Prevent the crons from re-downloading the same model over and over again
03b1cee
@RobertLucian
Fix models not getting reloaded if the timestamp is older
3b7004d
@RobertLucian
Merge branch 'master' into feature/multi-model-caching
101641f
@RobertLucian
Remove slipped comment
1fe6b7f
@RobertLucian RobertLucian merged commit 1cd5c22 into master Nov 11, 2020
@RobertLucian RobertLucian deleted the feature/multi-model-caching branch November 11, 2020 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deliahu deliahu Awaiting requested review from deliahu

1 more reviewer

@vishalbollu vishalbollu vishalbollu approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
enhancement New feature or request performance A performance improvement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support live reloading of models Serve a collection of custom models based on LRU
3 participants
@RobertLucian @vishalbollu @deliahu