[ML] Improve change point detection for long bucket lengths #95

tveasey · 2018-05-16T12:39:28Z

This PR does some hardening to change detection based on further testing, particularly for longer bucket lengths. By way of example, the following is the current behaviour on a problematic data set:
30m bucketing

1h bucketing

2h bucketing

After these changes we get:
30m bucketing

1h bucketing

2h bucketing

I have committed these changes incrementally in logical groups, so it may be easier to review them a commit at a time.

The following are made in the last commit:

Improve the methodology for deciding if we'll eventually accept a change. Specifically, we can observe all the values for the decision function over the change detection window and use these to predict whether we'll eventually accept the change. We use this to:
- Better adjust the weights of values added to the trend during change period,
- Better decide if it is worth continuing to test.
Improve the decision to clear the data structures we use for periodicity testing based on the historical data characteristics and the nature of the change.
Require a longer period of unusual values to trigger change detection. I've traded this for a slightly higher threshold on the individual p-values. There are downsides to testing for a change: i) it temporarily increases the model memory size, ii) we don't have concurrent change detection, so if the test is already running, it can interfere with detection if the test window overlaps a real change point. On balance, it is therefore better to wait slightly longer to see if the event is short-lived before committing to run the full test.
Decrease the time we'll take to detect a change for long bucket lengths. We can get acceptable test accuracy observing fewer values and with the current parameterisation we were waiting too long for a common case of scaling daily periodicity if the job used long bucket lengths.

Making these changes also uncovered an improvement to make to periodicity testing following on from #90. Specifically, we should have been weighting the samples when computing the level for the amplitude test to better deal with outliers. This is made in the first commit.

This can affect results for data with change points particularly if using longer bucket lengths. Marking as a non-issue since this is a refinement to an unreleased version.

…cket lengths)

droberts195 · 2018-05-23T10:23:32Z

lib/maths/CTimeSeriesChangeDetector.cc

    double p{CTools::logisticFunction(x[0], 0.05, 1.0) *
             CTools::logisticFunction(x[1], 0.1, 1.0) *
             (x[2] < 0.0 ? 1.0 : CTools::logisticFunction(x[2], 0.2, 1.0)) *
             CTools::logisticFunction(x[3], 0.2, 0.5)};
-    LOG_TRACE("p(" << (*candidates[0].second)->change()->print() << ") = " << p
-                   << " | x = " << core::CContainerPrinter::print(x));
+    LOG_TRACE(<< "df(" << (*candidates[0].second)->change()->print() << ") = " << p / 0.03125


It would make it clearer to the reader to use MAXIMUM_DECISION_FUNCTION * p rather than p / 0.03125 even though they evaluate to the same thing.

droberts195 · 2018-05-23T10:25:59Z

lib/maths/CTimeSeriesDecompositionDetail.cc

-        m_Windows[test]->initialize(time);
+void CTimeSeriesDecompositionDetail::CPeriodicityTest::maybeClear(core_t::TTime time,
+                                                                  double shift) {
+    for (auto& test : {E_Short, E_Long}) {


Could it just be auto for iterating simple enum values?

This was an oversight from originally iterating over the windows rather than the enums

droberts195 · 2018-05-23T10:27:31Z

lib/maths/CTimeSeriesDecompositionDetail.cc

+                    values.push_back(CBasicStatistics::mean(value));
+                }
+            }
+            if (shift > 1.4826 * CBasicStatistics::mad(values)) {


The magic 1.4826 could be a symbolic constant or at least commented.

This corrects for bias when estimating the standard deviation (for normal data). I've pulled out the constant as you suggest.

droberts195

LGTM

Backport #95.

do not clear periodicity test after detecting a changepoint: Internal QA test revealed a problem with deleting the periodicity test after detecting a changepoint, causing a regression in an internal dataset. This fix removes the clearance part of changepoint detection (only a small part of cp detection, see #95), effectively restoring the old behavior.

…stic#159) do not clear periodicity test after detecting a changepoint: Internal QA test revealed a problem with deleting the periodicity test after detecting a changepoint, causing a regression in an internal dataset. This fix removes the clearance part of changepoint detection (only a small part of cp detection, see elastic#95), effectively restoring the old behavior.

…#160) do not clear periodicity test after detecting a changepoint: Internal QA test revealed a problem with deleting the periodicity test after detecting a changepoint, causing a regression in an internal dataset. This fix removes the clearance part of changepoint detection (only a small part of cp detection, see #95), effectively restoring the old behavior.

tveasey added 4 commits May 16, 2018 10:34

Use weighted mean to compute level for amplitude test

50fad25

Median absolute deviation implementation

73dc9ac

Correct comment

a05ef09

Hardening improvements for change detection (particularly for long bu…

04ede41

…cket lengths)

tveasey added >enhancement v7.0.0 >non-issue :ml v6.4.0 review labels May 16, 2018

droberts195 reviewed May 23, 2018

View reviewed changes

tveasey added 2 commits May 25, 2018 11:32

Merge branch 'master' into enhancement/long-bucket-change-detection

b3d2026

Review comments

5211ecc

droberts195 approved these changes May 25, 2018

View reviewed changes

tveasey merged commit 943e3cf into elastic:master May 25, 2018

tveasey added a commit to tveasey/ml-cpp-1 that referenced this pull request May 29, 2018

[ML] Improve change point detection for long bucket lengths (elastic#95)

1040889

tveasey mentioned this pull request May 30, 2018

[6.x][ML] Improve change point detection for long bucket lengths #109

Merged

tveasey added a commit that referenced this pull request May 30, 2018

[6.x][ML] Improve change point detection for long bucket lengths (#109)

91bd889

Backport #95.

sophiec20 added :ml and removed :ml labels Jun 12, 2018

hendrikmuhs mentioned this pull request Jul 19, 2018

[ML] do not clear periodicity test after detecting a changepoint #159

Merged

tveasey deleted the enhancement/long-bucket-change-detection branch May 1, 2019 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ML] Improve change point detection for long bucket lengths #95

[ML] Improve change point detection for long bucket lengths #95

Uh oh!

tveasey commented May 16, 2018 •

edited

Loading

Uh oh!

droberts195 May 23, 2018

Uh oh!

tveasey May 25, 2018

Uh oh!

droberts195 May 23, 2018

Uh oh!

tveasey May 25, 2018

Uh oh!

droberts195 May 23, 2018

Uh oh!

tveasey May 25, 2018

Uh oh!

droberts195 left a comment

Uh oh!

Uh oh!

[ML] Improve change point detection for long bucket lengths #95

[ML] Improve change point detection for long bucket lengths #95

Uh oh!

Conversation

tveasey commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

droberts195 May 23, 2018

Choose a reason for hiding this comment

Uh oh!

tveasey May 25, 2018

Choose a reason for hiding this comment

Uh oh!

droberts195 May 23, 2018

Choose a reason for hiding this comment

Uh oh!

tveasey May 25, 2018

Choose a reason for hiding this comment

Uh oh!

droberts195 May 23, 2018

Choose a reason for hiding this comment

Uh oh!

tveasey May 25, 2018

Choose a reason for hiding this comment

Uh oh!

droberts195 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tveasey commented May 16, 2018 •

edited

Loading