Skip to content

Add data drift method Mann-Whitney U test #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,8 +284,8 @@ The currently implemented detectors are listed in the following table.
<td style="text-align: center; border: 1px solid grey; padding: 8px;"><a href="https://doi.org/10.1007/978-3-540-75488-6_27">Nishida and Yamauchi (2007)</a></td>
</tr>
<tr>
<td rowspan="14" style="text-align: center; border: 1px solid grey; padding: 8px;">Data drift</td>
<td rowspan="12" style="text-align: center; border: 1px solid grey; padding: 8px;">Batch</td>
<td rowspan="15" style="text-align: center; border: 1px solid grey; padding: 8px;">Data drift</td>
<td rowspan="13" style="text-align: center; border: 1px solid grey; padding: 8px;">Batch</td>
<td rowspan="8" style="text-align: center; border: 1px solid grey; padding: 8px;">Distance based</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">U</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">N</td>
Expand Down Expand Up @@ -335,7 +335,7 @@ The currently implemented detectors are listed in the following table.
<td style="text-align: center; border: 1px solid grey; padding: 8px;"><a href="https://doi.org/10.1057/jors.2008.144">Wu and Olson (2010)</a></td>
</tr>
<tr>
<td rowspan="4" style="text-align: center; border: 1px solid grey; padding: 8px;">Statistical test</td>
<td rowspan="5" style="text-align: center; border: 1px solid grey; padding: 8px;">Statistical test</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">U</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">C</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">Chi-square test</td>
Expand All @@ -353,6 +353,12 @@ The currently implemented detectors are listed in the following table.
<td style="text-align: center; border: 1px solid grey; padding: 8px;">Kolmogorov-Smirnov test</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;"><a href="https://doi.org/10.2307/2280095">Massey Jr (1951)</a></td>
</tr>
<tr>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">U</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">N</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">Mann-Whitney U test</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;"><a href="https://doi.org/10.1214/aoms/1177730491">Mann and Whitney (1947)</a></td>
</tr>
<tr>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">U</td>
<td style="text-align: center; border: 1px solid grey; padding: 8px;">N</td>
Expand Down
1 change: 1 addition & 0 deletions docs/source/api_reference/detectors/data_drift/batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,5 +50,6 @@ The {mod}`frouros.detectors.data_drift.batch` module contains batch data drift d
ChiSquareTest
CVMTest
KSTest
MannWhitneyUTest
WelchTTest
```
2 changes: 2 additions & 0 deletions frouros/detectors/data_drift/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
KL,
KSTest,
PSI,
MannWhitneyUTest,
MMD,
WelchTTest,
)
Expand All @@ -29,6 +30,7 @@
"KL",
"KSTest",
"PSI",
"MannWhitneyUTest",
"MMDStreaming",
"WelchTTest",
]
2 changes: 2 additions & 0 deletions frouros/detectors/data_drift/batch/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
ChiSquareTest,
CVMTest,
KSTest,
MannWhitneyUTest,
WelchTTest,
)

Expand All @@ -28,6 +29,7 @@
"KL",
"KSTest",
"PSI",
"MannWhitneyUTest",
"MMD",
"WelchTTest",
]
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@
from .chisquare import ChiSquareTest
from .cvm import CVMTest
from .ks import KSTest
from .mann_whitney_u import MannWhitneyUTest
from .welch_t_test import WelchTTest

__all__ = [
"ChiSquareTest",
"CVMTest",
"KSTest",
"MannWhitneyUTest",
"WelchTTest",
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Mann-Whitney U test module."""

from typing import Optional, List, Union

import numpy as np # type: ignore
from scipy.stats import mannwhitneyu # type: ignore

from frouros.callbacks.batch.base import BaseCallbackBatch
from frouros.detectors.data_drift.base import NumericalData, UnivariateData
from frouros.detectors.data_drift.batch.statistical_test.base import (
BaseStatisticalTest,
StatisticalResult,
)


class MannWhitneyUTest(BaseStatisticalTest):
"""Mann–Whitney U test [mann1947test]_ detector.

:References:

.. [mann1947test] Mann, Henry B., and Donald R. Whitney.
"On a test of whether one of two random variables is stochastically larger than
the other."
The annals of mathematical statistics (1947): 50-60.
"""

def __init__(
self,
callbacks: Optional[Union[BaseCallbackBatch, List[BaseCallbackBatch]]] = None,
) -> None:
"""Init method.

:param callbacks: callbacks
:type callbacks: Optional[Union[BaseCallbackBatch, List[BaseCallbackBatch]]]
"""
super().__init__(
data_type=NumericalData(),
statistical_type=UnivariateData(),
callbacks=callbacks,
)

def _statistical_test(
self, X_ref: np.ndarray, X: np.ndarray, **kwargs # noqa: N803
) -> StatisticalResult:
test = mannwhitneyu( # pylint: disable=unexpected-keyword-arg
x=X_ref,
y=X,
alternative="two-sided",
nan_policy="raise",
**kwargs,
)
test = StatisticalResult(
statistic=test.statistic,
p_value=test.pvalue,
)
return test
2 changes: 2 additions & 0 deletions frouros/tests/integration/test_data_drift.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
ChiSquareTest,
CVMTest,
KSTest,
MannWhitneyUTest,
WelchTTest,
)
from frouros.detectors.data_drift.batch.base import BaseDataDriftBatch
Expand Down Expand Up @@ -161,6 +162,7 @@ def test_batch_distance_bins_based_univariate_same_distribution(
[
(CVMTest(), 3776.09848103, 5.38105056e-07),
(KSTest(), 0.99576271, 0.0),
(MannWhitneyUTest(), 6912.0, 0.0),
(WelchTTest(), -287.92032554, 0.0),
],
)
Expand Down