Skip to content

Docs site rework for template functionality #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 22, 2023
Merged

Docs site rework for template functionality #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f4bbb7e
WIP docs rework for templates
nrichers Jun 20, 2023
80f7b08
Add text placeholder
nrichers Jun 20, 2023
603f56b
Re-add when to use test plans & tests
nrichers Jun 21, 2023
4d5cee6
Update rendered site
nrichers Jun 21, 2023
f37fbde
Merge branch 'main' of github.com:validmind/documentation into nriche…
nrichers Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion site/_quarto.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,13 @@ website:
# - guide/model-object.qmd
- text: "Testing"
contents:
- text: "Run a template"
- text: "Integrate a custom test provider"
- notebooks/how_to/implementing_custom_tests.ipynb
- notebooks/how_to/explore_test_suites_test_plans_and_tests.ipynb
- notebooks/how_to/run_a_test.ipynb
- notebooks/how_to/run_a_test_plan.ipynb
- notebooks/how_to/run_a_test_suite.ipynb
- notebooks/how_to/implementing_custom_tests.ipynb
- text: "---"
- text: "Jupyter notebooks"
file: guide/jupyter-notebooks.qmd
Expand Down
58 changes: 38 additions & 20 deletions site/_site/guide/developer-framework-introduction.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down Expand Up @@ -317,29 +323,41 @@ <h2 class="anchored" data-anchor-id="what-validminds-developer-framework-is">Wha
<h2 class="anchored" data-anchor-id="validmind-documentation-project-structure">ValidMind Documentation Project Structure</h2>
<!--- TO DO This looks more like glossary entries, should we move there? --->
<dl>
<dt>Projects</dt>
<dt>Project</dt>
<dd>
All documentation work in ValidMind is organized into projects which act as a container for the model documentation and validation report of your model. Each stage of the model’s MRM lifecycle will constitute a new project, and may be configured with its own templates and workflows.
</dd>
<dt>Model documentation</dt>
<dd>
A comprehensive record and description of a quantitative model. It should encompass all relevant information about the model in accordance with regulatory requirements (set by regulatory bodies) and model risk policies (set by an institution’s MRM team), assumptions, methodologies, data and inputs, model performance evaluation, limitations, and intended use. The purpose of model documentation is to provide transparency, facilitate understanding, and enable effective governance and oversight of the model.
Provides a comprehensive record and description of a quantitative model. This documentation should encompass all relevant information about the model in accordance with:
</dd>
<dt>Tests</dt>
<dd>
A function contained in the ValidMind Developer Framework, which is designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the relevant templates.
</dd>
<dt>Test plans</dt>
<dd>
A collection of many tests which are meant to be run simultaneously to validate and document specific aspects of the documentation. For instance, the <code>tabular_dataset</code> test plan runs several descriptive and data quality tests on a structured dataset, and documents the results in the ValidMind UI.
</dd>
<dt>Test suites</dt>
</dl>
<ul>
<li>Regulatory requirements (set by regulatory bodies)</li>
<li>Model risk policies (set by an institution’s MRM team)</li>
<li>Assumptions</li>
<li>Methodologies</li>
<li>Data and inputs</li>
<li>Model performance evaluation</li>
<li>Limitations</li>
<li>Intended use</li>
</ul>
<p>The purpose of model documentation is to provide transparency, facilitate understanding, and enable effective governance and oversight of the model.</p>
<dl>
<dt>Template</dt>
<dd>
Collection of test plans which are meant to run together to automate generate model documentation end-to-end for specific use-cases.
Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results:
</dd>
<dt>Templates</dt>
</dl>
<ul>
<li>Model documentation is populated when a user successfully executes all the tests contained in a template, thereby completing the test suite. Template placeholders get populated with content generated by the ValidMind Developer Framework.</li>
<li>Essentially, our platform scans these templates, identifies all tests and systematically organizes them into a well-structured test suite.</li>
</ul>
<p>The criteria for these templates are typically provided by your model risk management team. They can be programmatically customized to suit the unique requirements of each model use case, a task usually performed by an administrator.</p>
<dl>
<dt>Tests</dt>
<dd>
An outline of the sections/sub-sections of a ValidMind document (model documentation or validation report) and how they are organized. Templates also contain boilerplates and documentation &amp; test results placeholders for which content will be provided by the Developer Framework. Template requirements are typically provided by the model risk management team, and can be configured programmatically for each model use case, typically by an administrator.
A function contained in the ValidMind Developer Framework, which is designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the relevant templates.
</dd>
</dl>
<!---
Expand Down
18 changes: 12 additions & 6 deletions site/_site/guide/developer-framework.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/guide/install-and-initialize-developer-framework.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -239,12 +251,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/guide/jupyter-notebooks.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/guide/reference.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/guide/supported-models.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
25 changes: 15 additions & 10 deletions site/_site/guide/use-test-plans-and-tests.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -205,12 +217,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down Expand Up @@ -272,7 +278,7 @@
<h2 id="toc-title">On this page</h2>

<ul>
<li><a href="#what-tests-test-plans-and-test-suites-are" id="toc-what-tests-test-plans-and-test-suites-are" class="nav-link active" data-scroll-target="#what-tests-test-plans-and-test-suites-are">What Tests, Test plans, and Test suites are</a></li>
<li><a href="#what-tests-test-plans-and-test-suites-are" id="toc-what-tests-test-plans-and-test-suites-are" class="nav-link active" data-scroll-target="#what-tests-test-plans-and-test-suites-are">What tests, test plans, and test suites are</a></li>
<li><a href="#when-to-use-validmind-tests-test-plans-and-test-suites" id="toc-when-to-use-validmind-tests-test-plans-and-test-suites" class="nav-link" data-scroll-target="#when-to-use-validmind-tests-test-plans-and-test-suites">When to use ValidMind Tests, Test plans, and Test suites</a></li>
<li><a href="#api-reference" id="toc-api-reference" class="nav-link" data-scroll-target="#api-reference">API Reference</a></li>
</ul>
Expand Down Expand Up @@ -304,11 +310,10 @@ <h1 class="title">When to use test plans and tests</h1>
<li><p>When to use test plans and tests, and typical scenarios</p></li>
</ul>
<section id="what-tests-test-plans-and-test-suites-are" class="level2">
<h2 class="anchored" data-anchor-id="what-tests-test-plans-and-test-suites-are">What Tests, Test plans, and Test suites are</h2>
<h2 class="anchored" data-anchor-id="what-tests-test-plans-and-test-suites-are">What tests, test plans, and test suites are</h2>
<ul>
<li><p><strong>Tests</strong> are designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the relevant templates.</p></li>
<li><p><strong>Test plans</strong> are collections of tests which are meant to be run simultaneously to address specific aspects of the documentation.</p>
<p>Example: the <code>tabular_dataset</code> test plan runs several descriptive and data quality tests on a structured dataset, and documents the results in the ValidMind Platform.</p></li>
<li><p><strong>Test plans</strong> are collections of tests which are meant to be run simultaneously to address specific aspects of the documentation. quart Example: the <code>tabular_dataset</code> test plan runs several descriptive and data quality tests on a structured dataset, and documents the results in the ValidMind Platform.</p></li>
<li><p><strong>Test suites</strong> are collection of test plans which are meant to run together to automate generate model documentation end-to-end for specific use-cases.</p>
<p>Example: the <code>binary_classifier_full_suite</code> test suite runs the <code>tabular_dataset</code> and <code>binary_classifier</code> test plans to fully document the data and model sections for binary classification model use-cases.</p></li>
</ul>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/notebooks/Introduction_Customer_Churn.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -238,12 +250,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
18 changes: 12 additions & 6 deletions site/_site/notebooks/Quickstart_Customer Churn_full_suite.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,18 @@
</div>
<ul id="" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<span class="menu-text">Run a template</span>
</li>
<li class="sidebar-item">
<span class="menu-text">Integrate a custom test provider</span>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/explore_test_suites_test_plans_and_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Viewing all available Test suites, Test plans and tests</span></a>
Expand All @@ -238,12 +250,6 @@
<a href="../notebooks/how_to/run_a_test_suite.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Running an Individual Test Suite</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="../notebooks/how_to/implementing_custom_tests.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Implementing Custom Tests</span></a>
</div>
</li>
</ul>
</li>
Expand Down
Loading