merge: develop -> main (#260)

* chore: update files for new gitflow (#227) * chore: update files for new-release * chore: update files for new gitflow * chore: update readme (#229) * docs: update tutorials * docs: update tutorials (#230) * feat: new datasets and docs (#231) * feat: new datasets and docs * fix: fmt * docs: rename lib (#234) * fix: datasets labels (#245) * feature: update dataset_shift module (#248) * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * fix: update dataset_shift code and plots * fix: update dataset_shift code and plots * fix: update dataset_shift code and plots * fix: percent_degradation to ADP nomenclature * chore(deps): bump actions/checkout from 4.1.7 to 4.2.2 (#250) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.7 to 4.2.2. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.1.7...v4.2.2) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump actions/setup-python from 5.2.0 to 5.3.0 (#251) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.2.0 to 5.3.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5.2.0...v5.3.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: New Robustness Regression Attacker: Ridge Poissoner (#241) * feat: adding adaptive and randflip initializers for regression poisoners * feat: adding ridge-based regression poisoner * test: adding unit tests for ridge-based regression poisoner * chore: updating tutorial with ridge-based poisoner * docs: updating regression poisoner docs with ridge regressor * docs: adding docstrings for ridge regressor poisoner * chore: fixing lint * chore: updating docstrings for poisoners * chore: updloading notebook file for regression poisoner tutorial * docs: updating possible parameters in docstrings * feat: new methods for measuring security privacy risk score (#254) * feat: adding source code and tutorial for privacy risk score * tests: adding unit test for privacy risk score metric * docs: updating reference docstrings for privacy risk score metric * docs: updating docs for example gallery for security module * feat: adding tutorial for privacy risk score with different models * chore: adding tutorial for example gallery * chore: adding tutorial notebook files for docs section * Update security.rst --------- Co-authored-by: Cristian Muñoz <cristian.munoz@holisticai.com> * feat: new xai metrics based on globa feature importance, local feature importance and surrogate (#255) * internal changes * fix: pfi regression and shap * update: shap and contrast * chore: improved xai metrics * fix: paper regression results * ok * chore: xai-metrics updated * chore: format files * test: add test for new metrics * docs: update documentation and docstring format * chore: hatch dmt --------- Co-authored-by: Kleyton da Costa <kleyton.vsc@gmail.com> * chore: Improve Accuracy Degradation Profile Documentation (#249) * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * feature: update dataset_shift module * fix: update dataset_shift code and plots * fix: update dataset_shift code and plots * fix: update dataset_shift code and plots * fix: percent_degradation to ADP nomenclature * fix: uptade documentation * chore: clean adp tutorial * chore: remove inccorect reference * chore: format * doc: fix docstring --------- Co-authored-by: crismunoz <crisstrink@gmail.com> * docs:update tutorials (#257) * docs: updating docstrings for regression poissoners (#256) * fix: load dataset (#259) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Kleyton da Costa <44351707+Kleyt0n@users.noreply.github.com> Co-authored-by: andrelfnovaes <40208986+andrelfnovaes@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Franklin Cardenoso <44931423+fracarfer5@users.noreply.github.com> Co-authored-by: Kleyton da Costa <kleyton.vsc@gmail.com>
holistic-ai · Nov 1, 2024 · b0388cf · b0388cf
1 parent e87858c
commit b0388cf
Show file tree

Hide file tree

Showing 126 changed files with 20,580 additions and 6,274 deletions.
diff --git a/.github/workflows/feature-workflow.yml b/.github/workflows/feature-workflow.yml
@@ -16,10 +16,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4.1.7
+        uses: actions/checkout@v4.2.2
 
       - name: Set up Python
-        uses: actions/setup-python@v5.2.0
+        uses: actions/setup-python@v5.3.0
         with:
           python-version: "3.9"
           cache: "pip"

diff --git a/.github/workflows/publish-workflow.yml b/.github/workflows/publish-workflow.yml
@@ -14,15 +14,15 @@ jobs:
           private-key: ${{ secrets.APPLICATION_PRIVATE_KEY }}
 
       - name: Checkout code
-        uses: actions/checkout@v4.1.7
+        uses: actions/checkout@v4.2.2
         with:
           fetch-tags: true
           fetch-depth: 0
           ref: main
           token: ${{ steps.app-token.outputs.token }} # Needed to trigger other actions
 
       - name: Set up Python
-        uses: actions/setup-python@v5.2.0
+        uses: actions/setup-python@v5.3.0
         with:
             python-version: "3.9"
 

diff --git a/.github/workflows/release-workflow.yml b/.github/workflows/release-workflow.yml
@@ -20,15 +20,15 @@ jobs:
           private-key: ${{ secrets.APPLICATION_PRIVATE_KEY }}
 
       - name: Checkout code
-        uses: actions/checkout@v4.1.7
+        uses: actions/checkout@v4.2.2
         with:
           fetch-tags: true
           fetch-depth: 0
           ref: main
           token: ${{ steps.app-token.outputs.token }} # Needed to trigger other actions
 
       - name: Set up Python
-        uses: actions/setup-python@v5.2.0
+        uses: actions/setup-python@v5.3.0
         with:
           python-version: "3.9"
 

diff --git a/.gitignore b/.gitignore
@@ -36,4 +36,5 @@ docs/build
 .pytest_cache
 docs/**/*.ipynb
 src/*.ipynb
-.env
+.env
+paper
diff --git a/docs/source/_static/datasets.svg b/docs/source/_static/datasets.svg
diff --git a/docs/source/_static/explainability.svg b/docs/source/_static/explainability.svg
diff --git a/docs/source/_static/fairness.svg b/docs/source/_static/fairness.svg
diff --git a/docs/source/_static/robustness.svg b/docs/source/_static/robustness.svg
diff --git a/docs/source/_static/security.svg b/docs/source/_static/security.svg
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -67,13 +67,7 @@
     "sphinxcontrib.youtube",
 ]
 
-intersphinx_mapping = {
-    "python3": ("https://docs.python.org/3", None),
-    "numpy": ("https://numpy.org/doc/stable/", None),
-    "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
-    "sklearn": ("https://scikit-learn.org/stable/", None),
-    "matplotlib": ("https://matplotlib.org/stable/", None)
-}
+
 
 source_suffix = {
     '.rst': 'restructuredtext',
@@ -82,7 +76,7 @@
 
 
 nbsphinx_allow_errors = True  # Permitir errores en los notebooks
-nbsphinx_execute = 'auto'  # Puede ser 'auto', 'always', o 'never'
+nbsphinx_execute = 'never'  # Puede ser 'auto', 'always', o 'never'
 
 html_show_sourcelink = False
 # autodoc options

diff --git a/docs/source/gallery/index.rst b/docs/source/gallery/index.rst
@@ -4,13 +4,52 @@ Example Gallery
 
 This gallery contains a collection of examples that demonstrate the usage of ``holisticai`` library.
 
-.. toctree::
-    :maxdepth: 2
-
-    Datasets <tutorials/datasets>
+.. grid:: 1 2 2 2
+    :gutter: 4
+    :padding: 2 2 0 0
+    :class-container: sd-text-center
 
-    Bias <tutorials/bias>
+    .. grid-item-card:: Bias
+        :img-top: ../_static/fairness.svg
+        :class-card: intro-card
+        :link: tutorials/bias.html
+        :link-alt: tutorial_bias
+        :shadow: md
 
-    Explainability <tutorials/explainability>
+        Learn how to measure and mitigate bias in machine learning models
 
-    Robustness <tutorials/robustness>
+    .. grid-item-card:: Explainability
+        :img-top: ../_static/explainability.svg
+        :class-card: intro-card
+        :link: tutorials/explainability.html
+        :link-alt: reference
+        :shadow: md
+
+        Explain and quantify explainability of your machine learning model predictions
+
+    .. grid-item-card:: Robustness
+        :img-top: ../_static/robustness.svg
+        :class-card: intro-card
+        :link: tutorials/robustness.html
+        :link-alt: tutorials
+        :shadow: md
+
+        Evaluate your model robustness against adversarial attacks and data shifts
+
+    .. grid-item-card:: Security
+        :img-top: ../_static/security.svg
+        :class-card: intro-card
+        :link: tutorials/security.html
+        :link-alt: security
+        :shadow: md
+
+        Learn how to secure your machine learning models
+
+    .. grid-item-card:: Datasets
+        :img-top: ../_static/datasets.svg
+        :class-card: intro-card
+        :link: tutorials/datasets.html
+        :link-alt: datasets
+        :shadow: md
+
+        Explore and download datasets for your machine learning projects
diff --git a/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_classification.ipynb b/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_classification.ipynb
diff --git a/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_clustering.ipynb b/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_clustering.ipynb
diff --git a/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_multiclass.ipynb b/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_multiclass.ipynb
diff --git a/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_recommender.ipynb b/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_recommender.ipynb
diff --git a/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_regression.ipynb b/docs/source/gallery/tutorials/bias/measuring_bias/measuring_bias_regression.ipynb
diff --git a/docs/source/gallery/tutorials/explainability.rst b/docs/source/gallery/tutorials/explainability.rst
@@ -14,11 +14,17 @@ Measuring
 Here are some tutorials where different systems are assessed using the metrics.
 
 - Demos
-   - `Measuring XAI using Permutation Feature Importance <explainability/demos/global_permutation.ipynb>`_
-   - `Measuring XAI using Surrogate Feature Importance <explainability/demos/global_surrogate.ipynb>`_
-   - `Measuring XAI using SHAP Feature Importance <explainability/demos/local_shap.ipynb>`_
-   - `Measuring XAI using LIME Feature Importance <explainability/demos/local_lime.ipynb>`_
+   .. - `Measuring XAI using Permutation Feature Importance <explainability/demos/global_permutation.ipynb>`_
+   .. - `Measuring XAI using Surrogate Feature Importance <explainability/demos/global_surrogate.ipynb>`_
+   .. - `Measuring XAI using SHAP Feature Importance <explainability/demos/local_shap.ipynb>`_
+   .. - `Measuring XAI using LIME Feature Importance <explainability/demos/local_lime.ipynb>`_
+
+   - `Measuring XAI using Global Feature Importance <explainability/demos/xai_metrics_global_feature_importance.ipynb>`_
+   - `Measuring XAI using Local Feature Importance <explainability/demos/xai_metrics_local_feature_importance.ipynb>`_
+   - `Measuring XAI using Surrogate <explainability/demos/xai_metrics_surrogate.ipynb>`_
    - `Measuring XAI using Tree Based Metrics <explainability/demos/tree_based_metrics.ipynb>`_
 
 - Examples
-   - `Explainability in Machine Learning - Clinical Records Analysis <explainability/examples/example_clinical_records.ipynb>`_
+   - `Explainability in Machine Learning - Clinical Records Analysis <explainability/examples/example_clinical_records.ipynb>`_
+   - `Explainability in Regression  - US Crime <explainability/examples/measuring_explainability_us_crime.ipynb>`_
+   - `Explainability in Classification - Adult Dataset <explainability/examples/meauring_explainability_adult.ipynb>`_
diff --git a/docs/source/gallery/tutorials/explainability/demos/entropy_measures.ipynb b/docs/source/gallery/tutorials/explainability/demos/entropy_measures.ipynb
diff --git a/docs/source/gallery/tutorials/explainability/demos/global_permutation.ipynb b/docs/source/gallery/tutorials/explainability/demos/global_permutation.ipynb
diff --git a/docs/source/gallery/tutorials/explainability/demos/global_surrogate.ipynb b/docs/source/gallery/tutorials/explainability/demos/global_surrogate.ipynb
@@ -113,7 +113,7 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Rank Alignment</th>\n",
-       "      <td>0.250000</td>\n",
+       "      <td>0.083333</td>\n",
        "      <td>1.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
@@ -123,8 +123,8 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Spread Divergence</th>\n",
-       "      <td>0.060155</td>\n",
-       "      <td>0.0</td>\n",
+       "      <td>0.939845</td>\n",
+       "      <td>1.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Surrogate Accuracy Score</th>\n",
@@ -141,9 +141,9 @@
        "Alpha Importance Score    0.009901        0.0\n",
        "XAI Ease Score            1.000000        1.0\n",
        "Position Parity           0.250000        1.0\n",
-       "Rank Alignment            0.250000        1.0\n",
+       "Rank Alignment            0.083333        1.0\n",
        "Spread Ratio              0.147387        0.0\n",
-       "Spread Divergence         0.060155        0.0\n",
+       "Spread Divergence         0.939845        1.0\n",
        "Surrogate Accuracy Score  0.007099        1.0"
       ]
      },
@@ -221,7 +221,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 13,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:41.257635Z",
@@ -237,7 +237,7 @@
        "0.7957987838584853"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -260,7 +260,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 14,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:41.520942Z",
@@ -328,8 +328,8 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Spread Divergence</th>\n",
-       "      <td>0.043963</td>\n",
-       "      <td>0.0</td>\n",
+       "      <td>0.956037</td>\n",
+       "      <td>1.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Surrogate Accuracy Score</th>\n",
@@ -348,11 +348,11 @@
        "Position Parity           0.000000        1.0\n",
        "Rank Alignment            0.000000        1.0\n",
        "Spread Ratio              0.135284        0.0\n",
-       "Spread Divergence         0.043963        0.0\n",
+       "Spread Divergence         0.956037        1.0\n",
        "Surrogate Accuracy Score  0.980873        1.0"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -380,7 +380,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 15,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:42.755085Z",
@@ -396,7 +396,7 @@
        "<Axes: >"
       ]
      },
-     "execution_count": 8,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     },
@@ -426,7 +426,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 16,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:43.140051Z",
@@ -442,7 +442,7 @@
        "0.810126582278481"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -465,7 +465,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 17,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:43.190705Z",
@@ -523,7 +523,7 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Rank Alignment</th>\n",
-       "      <td>0.074074</td>\n",
+       "      <td>0.150000</td>\n",
        "      <td>1.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
@@ -533,8 +533,8 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Spread Divergence</th>\n",
-       "      <td>0.195874</td>\n",
-       "      <td>0.0</td>\n",
+       "      <td>0.804126</td>\n",
+       "      <td>1.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Surrogate Accuracy Score</th>\n",
@@ -551,13 +551,13 @@
        "Alpha Importance Score    0.115385        0.0\n",
        "XAI Ease Score            0.888889        1.0\n",
        "Position Parity           0.000000        1.0\n",
-       "Rank Alignment            0.074074        1.0\n",
+       "Rank Alignment            0.150000        1.0\n",
        "Spread Ratio              0.437464        0.0\n",
-       "Spread Divergence         0.195874        0.0\n",
+       "Spread Divergence         0.804126        1.0\n",
        "Surrogate Accuracy Score  0.848101        1.0"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -585,7 +585,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 18,
    "metadata": {
     "execution": {
      "iopub.execute_input": "2024-09-08T06:17:43.901406Z",
@@ -601,7 +601,7 @@
        "<Axes: >"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     },
@@ -625,7 +625,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "testing",
    "language": "python",
    "name": "python3"
   },