Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for online drift detectors #1108

Merged
merged 10 commits into from
Apr 20, 2023
Merged

Add support for online drift detectors #1108

merged 10 commits into from
Apr 20, 2023

Conversation

ascillitoe
Copy link
Contributor

@ascillitoe ascillitoe commented Apr 19, 2023

This PR adds support for online drift detectors to Alibi Detect runtime.

Overview

This involves two primary changes:

  1. Online drift detectors are run on one instance at a time, with the time-step self.t updated each time. If a request contains a batch of instances, the payload must be unpacked and passed to the detector one instance at a time.
  2. Online drift detectors have state, which must be periodically saved for checkpointing. State is saved every state_save_freq time-steps, where state_save_freq is a new configurable parameter added to AlibiDetectSettings.

Limitations

Parallel inference is NOT currently supported for online detectors (i.e. parallel_workers=0 is required). This would require careful thought wrt to how state is managed/shared across parallel workers, in addition to considering the implications of distributing non-iid samples between workers.

TODO's

# If batch is configured, wrap X in a list so that it is not unpacked
X = np.array(input_data)
if self._ad_settings.batch_size:
X = [X]
Copy link
Contributor Author

@ascillitoe ascillitoe Apr 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap X in a list (of length 1) so that we don't unpack into single instances when running in batch mode (i.e. offline drift detectors or outlier detectors). pred is then returned as a list of length 1.

This slightly circuitous logic is intended to avoid having two duplicate try-except blocks for the two (offline vs online) use cases...

Copy link
Contributor

@adriangonz adriangonz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @ascillitoe! 🚀

I've added a minor comment around the state save logic and check, but everything else should be good to go once the linter and tests pass!

runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated Show resolved Hide resolved
runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated Show resolved Hide resolved
runtimes/alibi-detect/tests/conftest.py Outdated Show resolved Hide resolved
@SachinVarghese SachinVarghese self-requested a review April 19, 2023 17:25
@ascillitoe ascillitoe marked this pull request as ready for review April 20, 2023 12:05
@adriangonz adriangonz merged commit 1e7bc4b into SeldonIO:master Apr 20, 2023
adriangonz pushed a commit that referenced this pull request Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants