Skip to content

Delete finding docs update #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions site/_quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,8 @@ website:
contents:
- text: "View and filter findings"
file: guide/model-validation/view-filter-model-findings.qmd
- text: "Add and update findings"
file: guide/model-validation/add-update-model-findings.qmd
- text: "Add and manage findings"
file: guide/model-validation/add-manage-model-findings.qmd
- guide/model-validation/view-reports.qmd
- guide/model-documentation/export-documentation.qmd
- text: "---"
Expand Down
4 changes: 2 additions & 2 deletions site/guide/model-validation/_assess-compliance-assess.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

![](/guide/model-validation/assess-compliance.png){width=60% fig-alt="A screenshot of the validation report section 2.1.1. that shows a compliance assessment with the option to link to finding"}

1. On the **Link Finding to Report** page that opens, select from the list of available findings, or create a new finding.^[[Create new finding](/guide/model-validation/add-update-model-findings.qmd)]
1. On the **Link Finding to Report** page that opens, select from the list of available findings, or create a new finding.^[[Create new finding](/guide/model-validation/add-manage-model-findings.qmd)]

1. Under **Risk Assessment Notes**, add any relevant notes that explain your assessment further.

Expand Down Expand Up @@ -51,7 +51,7 @@ A compliance summary gets shown for each subsection under **2. Validation** and

![](/guide/model-validation/assess-compliance.png){width=60% fig-alt="A screenshot of the validation report section 2.1.1. that shows a compliance assessment with the option to link to finding"}

1. On the **Link Finding to Report** page that opens, select from the list of available findings, or [create a new finding](/guide/model-validation/add-update-model-findings.qmd).
1. On the **Link Finding to Report** page that opens, select from the list of available findings, or [create a new finding](/guide/model-validation/add-manage-model-findings.qmd).

1. Under **Risk Assessment Notes**, add any relevant notes that explain your assessment further.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

![](/guide/model-validation/link-finding.png){width=60% fig-alt="A screenshot of the validation report section 2.1.1. that shows a compliance assessment with the option to link to finding"}

1. On the **Link Finding to Report** page that opens, select from the list of available findings, or create a new finding.^[[Create new finding](/guide/model-validation/add-update-model-findings.qmd)]
1. On the **Link Finding to Report** page that opens, select from the list of available findings, or create a new finding.^[[Create new finding](/guide/model-validation/add-manage-model-findings.qmd)]

1. Click **Update Linked Findings**.

Expand All @@ -34,7 +34,7 @@ The newly linked-to finding now gets shown under **Findings**.

![](/guide/model-validation/link-finding.png){width=60% fig-alt="A screenshot of the validation report section 2.1.1. that shows a compliance assessment with the option to link to finding"}

1. On the **Link Finding to Report** page that opens, select from the list of available findings, or [create a new finding](/guide/model-validation/add-update-model-findings.qmd).
1. On the **Link Finding to Report** page that opens, select from the list of available findings, or [create a new finding](/guide/model-validation/add-manage-model-findings.qmd).

1. Click **Update Linked Findings**.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
title: "Add and update model findings"
title: "Add and manage model findings"
date: last-modified
---

Add findings within the {{< var validmind.platform >}} at the model or documentation section level, and update your findings to include proposed remediation plans and supporting documentation.
Add findings within the {{< var validmind.platform >}} at the model or documentation section level, update your findings to include proposed remediation plans and supporting documentation, or delete findings that no longer need to be tracked.

::: {.prereq}

Expand Down Expand Up @@ -120,6 +120,22 @@ Uploaded files must be less than 50 MB in size.

![](delete-finding-attachment.gif){width=80% fig-alt="An animated gif demonstrating how to delete an attachment from a finding"}

## Delete model findings

::: {.callout-important title="Finding deletion is permanent."}
While finding deletion will be logged under your Model Activity,[^8] deleted findings cannot be retrieved.
:::

If you logged a finding in error or otherwise no longer need to track that finding, you can delete it:

1. Locate the finding you want to delete.[^9]

2. On the finding's detail page, click **{{< fa trash-can>}} Delete Finding** in the right sidebar.

Once you confirm, the finding will be permanently deleted.

![](delete-model-finding.gif){width=80% fig-alt="An animated gif demonstrating how to delete a finding"}


<!-- FOOTNOTES -->

Expand All @@ -135,4 +151,8 @@ Uploaded files must be less than 50 MB in size.

[^6]: [View and filter model findings](view-filter-model-findings.qmd)

[^7]: [Manage supporting documentation](#manage-supporting-documentation)
[^7]: [Manage supporting documentation](#manage-supporting-documentation)

[^8]: [View documentation activity](/guide/model-documentation/view-documentation-activity.qmd)

[^9]: [View and filter model findings](view-filter-model-findings.qmd)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions site/guide/model-validation/view-filter-model-findings.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,8 @@ Filters can be removed by clicking on the {{< fa xmark >}} next to them on the m

[^4]: [Filter model findings](#filter-model-findings)

[^5]: [Manage supporting documentation](add-update-model-findings.qmd#manage-supporting-documentation)
[^5]: [Manage supporting documentation](add-manage-model-findings.qmd#manage-supporting-documentation)

[^6]: [Working with the model inventory](/guide/model-inventory/working-with-model-inventory.qmd#search-filter-and-sort-models)

[^7]: [Manage supporting documentation](add-update-model-findings.qmd#manage-supporting-documentation)
[^7]: [Manage supporting documentation](add-manage-model-findings.qmd#manage-supporting-documentation)
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ listing:
fields: [title, description]
contents:
- view-filter-model-findings.qmd
- add-update-model-findings.qmd
- add-manage-model-findings.qmd
filters:
- tachyons
---
Expand Down
2 changes: 1 addition & 1 deletion site/python-docs/search.js

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions site/python-docs/validmind.html
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ <h1 class="modulename">
<section id="__version__">
<div class="attr variable">
<span class="name">__version__</span> =
<span class="default_value">&#39;2.5.7&#39;</span>
<span class="default_value">&#39;2.5.13&#39;</span>


</div>
Expand Down Expand Up @@ -685,7 +685,7 @@ <h6 id="arguments">Arguments:</h6>

<ul>
<li><strong>results (list):</strong> A list of ThresholdTestResults objects</li>
<li><strong>inputs (list):</strong> A list of input keys (names) that were used to run the test</li>
<li><strong>inputs (list):</strong> A list of input IDs that were used to run the test</li>
</ul>

<h6 id="raises">Raises:</h6>
Expand Down
4 changes: 2 additions & 2 deletions site/python-docs/validmind/tests.html

Large diffs are not rendered by default.

5 changes: 0 additions & 5 deletions site/python-docs/validmind/tests/data_validation.html
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,10 @@ <h3>Submodules</h3>
<ul>
<li><a href="data_validation/ACFandPACFPlot.html">ACFandPACFPlot</a></li>
<li><a href="data_validation/ADF.html">ADF</a></li>
<li><a href="data_validation/ANOVAOneWayTable.html">ANOVAOneWayTable</a></li>
<li><a href="data_validation/AutoAR.html">AutoAR</a></li>
<li><a href="data_validation/AutoMA.html">AutoMA</a></li>
<li><a href="data_validation/AutoSeasonality.html">AutoSeasonality</a></li>
<li><a href="data_validation/AutoStationarity.html">AutoStationarity</a></li>
<li><a href="data_validation/BivariateFeaturesBarPlots.html">BivariateFeaturesBarPlots</a></li>
<li><a href="data_validation/BivariateHistograms.html">BivariateHistograms</a></li>
<li><a href="data_validation/BivariateScatterPlots.html">BivariateScatterPlots</a></li>
<li><a href="data_validation/ChiSquaredFeaturesTable.html">ChiSquaredFeaturesTable</a></li>
<li><a href="data_validation/ClassImbalance.html">ClassImbalance</a></li>
Expand All @@ -51,7 +48,6 @@ <h3>Submodules</h3>
<li><a href="data_validation/Duplicates.html">Duplicates</a></li>
<li><a href="data_validation/EngleGrangerCoint.html">EngleGrangerCoint</a></li>
<li><a href="data_validation/FeatureTargetCorrelationPlot.html">FeatureTargetCorrelationPlot</a></li>
<li><a href="data_validation/HeatmapFeatureCorrelations.html">HeatmapFeatureCorrelations</a></li>
<li><a href="data_validation/HighCardinality.html">HighCardinality</a></li>
<li><a href="data_validation/HighPearsonCorrelation.html">HighPearsonCorrelation</a></li>
<li><a href="data_validation/IQROutliersBarPlot.html">IQROutliersBarPlot</a></li>
Expand All @@ -61,7 +57,6 @@ <h3>Submodules</h3>
<li><a href="data_validation/LaggedCorrelationHeatmap.html">LaggedCorrelationHeatmap</a></li>
<li><a href="data_validation/MissingValues.html">MissingValues</a></li>
<li><a href="data_validation/MissingValuesBarPlot.html">MissingValuesBarPlot</a></li>
<li><a href="data_validation/MissingValuesRisk.html">MissingValuesRisk</a></li>
<li><a href="data_validation/PearsonCorrelationMatrix.html">PearsonCorrelationMatrix</a></li>
<li><a href="data_validation/PhillipsPerronArch.html">PhillipsPerronArch</a></li>
<li><a href="data_validation/RollingStatsPlot.html">RollingStatsPlot</a></li>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,27 +75,29 @@ <h1 class="modulename">
<div class="docstring"><p>Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to
reveal trends and correlations.</p>

<p><strong>Purpose</strong>: The ACF (Autocorrelation Function) and PACF (Partial Autocorrelation Function) plot test is employed
to analyze time series data in machine learning models. It illuminates the correlation of the data over time by
plotting the correlation of the series with its own lags (ACF), and the correlations after removing effects already
accounted for by earlier lags (PACF). This information can identify trends, such as seasonality, degrees of
autocorrelation, and inform the selection of order parameters for AutoRegressive Integrated Moving Average (ARIMA)
models.</p>
<h3 id="purpose">Purpose</h3>

<p><strong>Test Mechanism</strong>: The <code><a href="#ACFandPACFPlot">ACFandPACFPlot</a></code> test accepts a dataset with a time-based index. It first confirms the
index is of a datetime type, then handles any NaN values. The test subsequently generates ACF and PACF plots for
each column in the dataset, producing a subplot for each. If the dataset doesn't include key columns, an error is
returned.</p>
<p>The ACF (Autocorrelation Function) and PACF (Partial Autocorrelation Function) plot test is employed to analyze
time series data in machine learning models. It illuminates the correlation of the data over time by plotting the
correlation of the series with its own lags (ACF), and the correlations after removing effects already accounted
for by earlier lags (PACF). This information can identify trends, such as seasonality, degrees of autocorrelation,
and inform the selection of order parameters for AutoRegressive Integrated Moving Average (ARIMA) models.</p>

<p><strong>Signs of High Risk</strong>:</p>
<h3 id="test-mechanism">Test Mechanism</h3>

<p>The <code><a href="#ACFandPACFPlot">ACFandPACFPlot</a></code> test accepts a dataset with a time-based index. It first confirms the index is of a datetime
type, then handles any NaN values. The test subsequently generates ACF and PACF plots for each column in the
dataset, producing a subplot for each. If the dataset doesn't include key columns, an error is returned.</p>

<h3 id="signs-of-high-risk">Signs of High Risk</h3>

<ul>
<li>Sudden drops in the correlation at a specific lag might signal a model at high risk.</li>
<li>Consistent high correlation across multiple lags could also indicate non-stationarity in the data, which may
suggest that a model estimated on this data won't generalize well to future, unknown data.</li>
</ul>

<p><strong>Strengths</strong>:</p>
<h3 id="strengths">Strengths</h3>

<ul>
<li>ACF and PACF plots offer clear graphical representations of the correlations in time series data.</li>
Expand All @@ -105,7 +107,7 @@ <h1 class="modulename">
parameters.</li>
</ul>

<p><strong>Limitations</strong>:</p>
<h3 id="limitations">Limitations</h3>

<ul>
<li>ACF and PACF plots are exclusively for time series data and hence, can't be applied to all ML models.</li>
Expand Down
44 changes: 24 additions & 20 deletions site/python-docs/validmind/tests/data_validation/ADF.html
Original file line number Diff line number Diff line change
Expand Up @@ -81,39 +81,43 @@ <h1 class="modulename">

<div class="docstring"><p>Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.</p>

<p><strong>Purpose</strong>: The Augmented Dickey-Fuller (ADF) test metric is used here to determine the order of integration,
i.e., the stationarity of a given time series data. The stationary property of data is pivotal in many machine
learning models as it impacts the reliability and effectiveness of predictions and forecasts.</p>
<h3 id="purpose">Purpose</h3>

<p><strong>Test Mechanism</strong>: The ADF test starts by executing the ADF function from the statsmodels library on every feature
of the dataset. Multiple outputs are generated for each run, including the ADF test statistic and p-value, count of
lags used, the number of observations factored into the test, critical values at various confidence levels, and the
maximized information criterion. These results are stored for each feature for subsequent analysis.</p>
<p>The Augmented Dickey-Fuller (ADF) test metric is used to determine the order of integration, i.e., the stationarity
of a given time series dataset. The stationary property of data is pivotal in many machine learning models as it
impacts the reliability and effectiveness of predictions and forecasts.</p>

<p><strong>Signs of High Risk</strong>:</p>
<h3 id="test-mechanism">Test Mechanism</h3>

<p>The ADF test is executed using the <code>adfuller</code> function from the <code>statsmodels</code> library on each feature of the
dataset. Multiple outputs are generated for each run, including the ADF test statistic and p-value, count of lags
used, the number of observations considered in the test, critical values at various confidence levels, and the
information criterion. These results are stored for each feature for subsequent analysis.</p>

<h3 id="signs-of-high-risk">Signs of High Risk</h3>

<ul>
<li>An inflated ADF statistic and high p-value (generally above 0.05) insinuate a high risk to the model's
performance due to the presence of a unit root indicating non-stationarity.</li>
<li>Such non-stationarity might result in untrustworthy or insufficient forecasts.</li>
<li>An inflated ADF statistic and high p-value (generally above 0.05) indicate a high risk to the model's performance
due to the presence of a unit root indicating non-stationarity.</li>
<li>Non-stationarity might result in untrustworthy or insufficient forecasts.</li>
</ul>

<p><strong>Strengths</strong>:</p>
<h3 id="strengths">Strengths</h3>

<ul>
<li>The ADF test is robust to more sophisticated correlation within the data, which empowers it to be deployed in
settings where data might display complex stochastic behavior.</li>
<li>The ADF test provides explicit outputs like test statistics, critical values, and information criterion, thereby
enhancing our understanding and transparency of the model validation process.</li>
<li>The ADF test is robust to sophisticated correlations within the data, making it suitable for settings where data
displays complex stochastic behavior.</li>
<li>It provides explicit outputs like test statistics, critical values, and information criterion, enhancing
understanding and transparency in the model validation process.</li>
</ul>

<p><strong>Limitations</strong>:</p>
<h3 id="limitations">Limitations</h3>

<ul>
<li>The ADF test might demonstrate low statistical power, making it challenging to differentiate between a unit root
and near-unit-root processes causing false negatives.</li>
<li>The test assumes the data follows an autoregressive process, which might not be the case all the time.</li>
<li>The ADF test finds it demanding to manage time series data with structural breaks.</li>
and near-unit-root processes, potentially causing false negatives.</li>
<li>It assumes the data follows an autoregressive process, which might not always be the case.</li>
<li>The test struggles with time series data that have structural breaks.</li>
</ul>
</div>

Expand Down
18 changes: 9 additions & 9 deletions site/python-docs/validmind/tests/data_validation/AutoAR.html
Original file line number Diff line number Diff line change
Expand Up @@ -77,42 +77,42 @@ <h1 class="modulename">

<div class="docstring"><p>Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.</p>

<p><strong>Purpose</strong>:</p>
<h3 id="purpose">Purpose</h3>

<p>The AutoAR test is intended to automatically identify the Autoregressive (AR) order of a time series by utilizing
the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). AR order is crucial in forecasting
tasks as it dictates the quantity of prior terms in the sequence to use for predicting the current term. The
objective is to select the most fitting AR model that encapsulates the trend and seasonality in the time series
data.</p>

<p><strong>Test Mechanism</strong>:</p>
<h3 id="test-mechanism">Test Mechanism</h3>

<p>The test mechanism operates by iterating through a possible range of AR orders up to a defined maximum. An AR model
is fitted for each order, and the corresponding BIC and AIC are computed. BIC and AIC statistical measures are
designed to penalize models for complexity, preferring simpler models that fit the data proficiently. To verify the
stationarity of the time series, the Augmented Dickey-Fuller test is executed. The AR order, BIC, and AIC findings,
stationarity of the time series, the Augmented Dickey-Fuller test is executed. The AR order, BIC, and AIC findings
are compiled into a dataframe for effortless comparison. Then, the AR order with the smallest BIC is established as
the desirable order for each variable.</p>

<p><strong>Signs of High Risk</strong>:</p>
<h3 id="signs-of-high-risk">Signs of High Risk</h3>

<ul>
<li>An augmented Dickey Fuller test p-value &gt; 0.05, indicating the time series isn't stationary, may lead to
inaccurate results.</li>
<li>Problems with the model fitting procedure, such as computational or convergence issues.</li>
<li>Continuous selection of the maximum specified AR order may suggest insufficient set limit.</li>
<li>Continuous selection of the maximum specified AR order may suggest an insufficient set limit.</li>
</ul>

<p><strong>Strengths</strong>:</p>
<h3 id="strengths">Strengths</h3>

<ul>
<li>The test independently pinpoints the optimal AR order, thereby reducing potential human bias.</li>
<li>It strikes a balance between model simplicity and goodness-of-fit to avoid overfitting.</li>
<li>Has the capability to account for stationarity in a time series, an essential aspect for dependable AR modelling.</li>
<li>The results are aggregated into an comprehensive table, enabling an easy interpretation.</li>
<li>Has the capability to account for stationarity in a time series, an essential aspect for dependable AR modeling.</li>
<li>The results are aggregated into a comprehensive table, enabling an easy interpretation.</li>
</ul>

<p><strong>Limitations</strong>:</p>
<h3 id="limitations">Limitations</h3>

<ul>
<li>The tests need a stationary time series input.</li>
Expand Down
Loading
Loading