Adding label distribution measurement

measurements/label_distribution/README.md

@@ -0,0 +1,95 @@
+---
+title: Label Distribution
+emoji: 🤗
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.0.2
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- measurement
+description: >-
+  Returns the label distribution and skew of the input data.
+---
+
+# Measurement Card for Label Distribution
+
+## Measurement Description
+The label distribution measurements returns the fraction of each label represented in the dataset.
+
+## Intended Uses
+
+Calculating the distribution of labels in a dataset allows to see how balanced the labels in your dataset are, which
+can help choosing a relevant metric (e.g. accuracy when the dataset is balanced, versus F1 score when there is an
+imbalance).
+
+## How to Use
+
+The measurement takes a list of labels as input:
+
+```python
+from evaluate import load
+>>> distribution = evaluate.load("label_distribution")
+>>> data = [1, 0, 2, 2, 0, 0, 0, 0, 0, 2]
+>>> results = distribution.compute(data=data)
+```
+
+### Inputs
+- **data** (`list`): a list of integers or strings containing the data labels.
+
+### Output Values
+By default, this metric outputs a dictionary that contains :
+-**label_distribution** (`dict`) : a dictionary containing two sets of keys and values: `labels`, which includes the list of labels contained in the dataset, and `fractions`, which includes the fraction of each label.
+-**label_skew** (`scalar`) : the asymmetry of the label distribution.
+
+```python
+{'label_distribution': {'labels': [1, 0, 2], 'fractions': [0.1, 0.6, 0.3]}, 'label_skew': 0.7417688338666573}
+```
+
+If skewness is 0, the dataset is perfectly balanced; if it is less than -1 or greater than 1, the distribution is highly skewed; anything in between can be considered moderately skewed.
+
+#### Values from Popular Papers
+
+
+### Examples
+Calculating the label distribution of a dataset with binary labels:
+
+```python
+>>> data = [1, 0, 1, 1, 0, 1, 0]
+>>> distribution = evaluate.load("label_distribution")
+>>> results = distribution.compute(data=data)
+>>> print(results)
+{'label_distribution': {'labels': [1, 0], 'fractions': [0.5714285714285714, 0.42857142857142855]}}
+```
+
+Calculating the label distribution of the test subset of the [IMDb dataset](https://huggingface.co/datasets/imdb):
+```python
+>>> from datasets import load_dataset
+>>> imdb = load_dataset('imdb', split = 'test')
+>>> distribution = evaluate.load("label_distribution")
+>>> results = distribution.compute(data=imdb['label'])
+>>> print(results)
+{'label_distribution': {'labels': [0, 1], 'fractions': [0.5, 0.5]}, 'label_skew': 0.0}
+```
+N.B. The IMDb dataset is perfectly balanced.
+
+The output of the measurement can easily be passed to matplotlib to plot a histogram of each label:
+
+```python
+>>> data = [1, 0, 2, 2, 0, 0, 0, 0, 0, 2]
+>>> distribution = evaluate.load("label_distribution")
+>>> results = distribution.compute(data=data)
+>>> plt.bar(results['label_distribution']['labels'], results['label_distribution']['fractions'])
+>>> plt.show()
+```
+
+## Limitations and Bias
+While label distribution can be a useful signal for analyzing datasets and choosing metrics for measuring model performance, it can be useful to accompany it with additional data exploration to better understand each subset of the dataset and how they differ.
+
+## Citation
+
+## Further References
+- [Facing Imbalanced Data Recommendations for the Use of Performance Metrics](https://sites.pitt.edu/~jeffcohn/skew/PID2829477.pdf)
+- [Scipy Stats Skew Documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html#scipy-stats-skew)

measurements/label_distribution/app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("label_distribution", module_type="measurement")
+launch_gradio_widget(module)

measurements/label_distribution/label_distribution.py

@@ -0,0 +1,93 @@
+# Copyright 2022 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Label Distribution Measurement."""
+
+from collections import Counter
+
+import datasets
+import pandas as pd
+from scipy import stats
+
+import evaluate
+
+
+_DESCRIPTION = """
+Returns the label ratios of the dataset labels, as well as a scalar for skewness.
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    `data`: a list containing the data labels
+
+Returns:
+    `label_distribution` (`dict`) :  a dictionary containing two sets of keys and values: `labels`, which includes the list of labels contained in the dataset, and `fractions`, which includes the fraction of each label.
+    `label_skew` (`scalar`) : the asymmetry of the label distribution.
+Examples:
+    >>> data = [1, 0, 1, 1, 0, 1, 0]
+    >>> distribution = evaluate.load("label_distribution")
+    >>> results = distribution.compute(data=data)
+    >>> print(results)
+    {'label_distribution': {'labels': [1, 0], 'fractions': [0.5714285714285714, 0.42857142857142855]}, 'label_skew': -0.2886751345948127}
+"""
+
+_CITATION = """\
+@ARTICLE{2020SciPy-NMeth,
+  author  = {Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and
+            Haberland, Matt and Reddy, Tyler and Cournapeau, David and
+            Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and
+            Bright, Jonathan and {van der Walt}, St{\'e}fan J. and
+            Brett, Matthew and Wilson, Joshua and Millman, K. Jarrod and
+            Mayorov, Nikolay and Nelson, Andrew R. J. and Jones, Eric and
+            Kern, Robert and Larson, Eric and Carey, C J and
+            Polat, {\.I}lhan and Feng, Yu and Moore, Eric W. and
+            {VanderPlas}, Jake and Laxalde, Denis and Perktold, Josef and
+            Cimrman, Robert and Henriksen, Ian and Quintero, E. A. and
+            Harris, Charles R. and Archibald, Anne M. and
+            Ribeiro, Ant{\^o}nio H. and Pedregosa, Fabian and
+            {van Mulbregt}, Paul and {SciPy 1.0 Contributors}},
+  title   = {{{SciPy} 1.0: Fundamental Algorithms for Scientific
+            Computing in Python}},
+  journal = {Nature Methods},
+  year    = {2020},
+  volume  = {17},
+  pages   = {261--272},
+  adsurl  = {https://rdcu.be/b08Wh},
+  doi     = {10.1038/s41592-019-0686-2},
+}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class LabelDistribution(evaluate.Measurement):
+    def _info(self):
+        return evaluate.MeasurementInfo(
+            module_type="measurement",
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=[
+                datasets.Features({"data": datasets.Value("int32")}),
+                datasets.Features({"data": datasets.Value("string")}),
+            ],
+        )
+
+    def _compute(self, data):
+        """Returns the fraction of each label present in the data"""
+        c = Counter(data)
+        label_distribution = {"labels": [k for k in c.keys()], "fractions": [f / len(data) for f in c.values()]}
+        if isinstance(data[0], str):
+            label2id = {label: id for id, label in enumerate(label_distribution["labels"])}
+            data = [label2id[d] for d in data]
+        skew = stats.skew(data)
+        return {"label_distribution": label_distribution, "label_skew": skew}

measurements/label_distribution/requirements.txt

@@ -0,0 +1,3 @@
+git+https://github.com/huggingface/evaluate@a45df1eb9996eec64ec3282ebe554061cb366388
+datasets~=2.0
+scipy