Skip to content

Merge main into staging #530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ These directories may have sub-directories depending on their size and grouped s
- `_site` — This is where static files rendered by `quarto render` get placed.
- `assets` — This is where general shared assets live (stylesheets, promotional images, all videos, etc.).
- `internal` — For internal testing only.
- `notebooks` — This is where notebooks retrieved from the [`developer-framework` repo](https://github.com/validmind/developer-framework) live.
- `tests` — This is where test descriptions generated from the Python source in the [developer-framework repo](https://github.com/validmind/developer-framework) live.
- `notebooks` — This is where notebooks retrieved from the [`validmind-library` repo](https://github.com/validmind/validmind-library) live.
- `tests` — This is where test descriptions generated from the Python source in the [`validmind-library` repo](https://github.com/validmind/validmind-library) live.

### Auxiliary `internal` directories

Expand All @@ -91,7 +91,7 @@ These directories may have sub-directories depending on their size and grouped s

### `.qmd`

Files for the docs site are created using [Quarto Markdown](https://quarto.org/docs/authoring/markdown-basics.html) (`.qmd`). These, along with any Jupyter Notebooks pulled in from `developer-framework`, get rendered into HTML files.
Files for the docs site are created using [Quarto Markdown](https://quarto.org/docs/authoring/markdown-basics.html) (`.qmd`). These, along with any Jupyter Notebooks pulled in from `validmind-library`, get rendered into HTML files.

#### Hyperlinks

Expand Down Expand Up @@ -125,7 +125,7 @@ If there are additional files that Quarto does not copy over automatically, plac

Notebooks (`.ipynb` files) are NOT edited via this `documentation` repo, as any changes will be overridden.

Changes need to be made in the [root repository `developer-framework`](https://github.com/validmind/developer-framework) and pulled into this one with:
Changes need to be made in the [root repository `validmind-library`](https://github.com/validmind/validmind-library) and pulled into this one with:

```bash
make get-source
Expand Down
8 changes: 4 additions & 4 deletions site/Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Define source and destination directories
SRC_DIR := _source/developer-framework
SRC_DIR := _source/validmind-library
DEST_DIR_NB := notebooks
DEST_DIR_PYTHON := _site/validmind
DEST_DIR_TESTS := tests
Expand All @@ -12,7 +12,7 @@ GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)
help:
@echo "Available targets:"
@echo " clean Remove the _source/ directory"
@echo " clone Clone the developer-framework repository into _source/"
@echo " clone Clone the validmind-library repository into _source/"
@echo " notebooks Copy Jupyter notebooks into notebooks/"
@echo " python-docs Copy the Python library docs into _site/validmind"
@echo " get-source Get all source files (clean, clone, notebooks, python-docs)"
Expand All @@ -33,7 +33,7 @@ clean:
# Clone the source repository we need
clone:
@echo "\nCloning source repos ..."
git clone -b main git@github.com:validmind/developer-framework.git $(SRC_DIR)
git clone -b main git@github.com:validmind/validmind-library.git $(SRC_DIR)

# Copy over Jupyter notebooks and supporting files
notebooks:
Expand Down Expand Up @@ -65,7 +65,7 @@ python-docs:

test-descriptions:
@echo "\nUpdating test descriptions source ..."
@cd _source/developer-framework && make install && poetry run python scripts/extract_descriptions.py validmind/tests
@cd _source/validmind-library && make install && poetry run python scripts/extract_descriptions.py validmind/tests
@cd ../../
@rm -rf $(DEST_DIR_TESTS)
@mkdir -p $(DEST_DIR_TESTS)
Expand Down
2 changes: 1 addition & 1 deletion site/about/contributing/validmind-community.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,6 @@ As a member of the {{< var vm.product >}} community, we invite you to be part of
Please note that all community contributions are subject to review by the {{< var vm.product >}} team. See our [software license agreement](/about/fine-print/license-agreement.qmd) for more details.

- **[Code samples](/developer/samples-jupyter-notebooks.qmd)** — Have a Jupyter Notebook that works perfectly with {{< var vm.product >}}, or covers a sample use case not already provided? Send it to us!
- **[{{< var validmind.developer >}}](https://github.com/validmind/developer-framework/)** — Live by the {{< var vm.developer >}}, and have an idea that will streamline the experience for others? Let us know!
- **[{{< var validmind.developer >}}](https://github.com/validmind/validmind-library/)** — Live by the {{< var vm.developer >}}, and have an idea that will streamline the experience for others? Let us know!
- **[Product documentation](https://github.com/validmind/documentation)** — Spot a typo, or have identified a gap in instruction you think would be beneficial for the community to fill? Submit an edit!

2 changes: 1 addition & 1 deletion site/about/glossary/_validmind.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
These two features are intertwined and work in tandem to help streamline your model lifecycle.

<span id="framework">{{< var validmind.developer >}} ({{< var vm.developer >}})</span>
: An open-source^[**{{< var vm.product >}} GitHub:** [developer-framework](https://github.com/validmind/developer-framework/)] suite of documentation tools and test suites designed to document models, test models for weaknesses, and identify overfit areas. Enables automating the generation of model documentation by uploading documentation and test results to the {{< var validmind.platform >}}.
: An open-source^[**{{< var vm.product >}} GitHub:** [`validmind-library`](https://github.com/validmind/validmind-library/)] suite of documentation tools and test suites designed to document models, test models for weaknesses, and identify overfit areas. Enables automating the generation of model documentation by uploading documentation and test results to the {{< var validmind.platform >}}.

<span id="platform">{{< var validmind.platform >}} ({{< var vm.platform >}})</span>
: A hosted multi-tenant architecture^[[Log into the {{< var validmind.platform >}}](/guide/configuration/log-in-to-validmind.qmd)] that includes the {{< var vm.product >}} cloud-based web interface, APIs, databases, documentation and validation engine, and various internal services.
2 changes: 1 addition & 1 deletion site/developer/get-started-validmind-library.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ The {{< var validmind.developer >}} provides a rich collection of documentation

<!-- Using the variable in alt text messes up the image display -->

![](validmind-ai-risk-platform.png){width=70% fig-alt="An image showing the two main components of ValidMind. The ValidMind Library that integrates with your existing developer environment, and the ValidMind Platform."}
![](/get-started/validmind-lifecycle.jpg){width=70% fig-alt="An image showing the two main components of ValidMind. The ValidMind Library that integrates with your existing developer environment, and the ValidMind Platform."}

{{< var vm.product >}} offers two primary methods for automating model documentation:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,4 @@ Version: 2.5.15

[^5]: [Install {{< var vm.product >}}](#install-validmind)

[^6]: **GitHub:** [validmind/developer-framework/validmind/&lowbar;&lowbar;version&lowbar;&lowbar;.py](https://github.com/validmind/developer-framework/blob/prod/validmind/__version__.py)
[^6]: **ValidMind GitHub:** [validmind-library/validmind/&lowbar;&lowbar;version&lowbar;&lowbar;.py](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py)
2 changes: 1 addition & 1 deletion site/developer/samples-jupyter-notebooks.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Our code samples, based on Jupyter Notebooks, showcase the capabilities and feat

::: {.w-30-ns .mt2 .pb3}

[{{< fa brands github >}} Access Notebooks on GitHub](https://github.com/validmind/developer-framework){.button target="_blank"}
[{{< fa brands github >}} Access Notebooks on GitHub](https://github.com/validmind/validmind-library){.button target="_blank"}

:::

Expand Down
Binary file removed site/developer/validmind-ai-risk-platform.png
Binary file not shown.
2 changes: 1 addition & 1 deletion site/get-started/developer/try-in-your-own-environment.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Start by retrieving our notebook samples from GitHub, or download them from our
3. Run the following command:

```bash
git clone https://github.com/validmind/developer-framework.git
git clone https://github.com/validmind/validmind-library.git
```

4. After the cloning process is complete, open `notebooks/code_samples/quickstart_customer_churn_full_suite.ipynb` in your developer environment and [run the notebook](#run-the-notebook).
Expand Down
2 changes: 1 addition & 1 deletion site/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ The **purpose-built platform** for model risk management teams to test, document
<script src="scripts/validsearch.js"></script>

[quickstart](get-started/developer/try-with-jupyterhub.qmd){.button-pink .cta}
[open-source software](https://github.com/validmind/developer-framework/){.button-white .cta}
[open-source software](https://github.com/validmind/validmind-library/){.button-white .cta}

:::

Expand Down
6 changes: 3 additions & 3 deletions site/releases/2023/2023-nov-09/highlights.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This release introduces support for several new models, a new user onboarding gu
### {{< var validmind.developer >}} (v1.23.0)

<!--- NR the next three notebooks have been commented out as they are not ready to be released.--->
<!---[SC-2284] Support regression models by @AnilSorathiya in [#259](https://github.com/validmind/developer-framework/pull/259)--->
<!---[SC-2284] Support regression models by @AnilSorathiya in [#259](https://github.com/validmind/validmind-library/pull/259)--->
#### Support for regression models

The {{< var validmind.developer >}} has added support for regression models. The updates include:
Expand All @@ -23,7 +23,7 @@ The {{< var validmind.developer >}} has added support for regression models. The

<!---[Try it ...](/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb)--->

<!---[SC-2411] Clustering models support by @AnilSorathiya in [#271](https://github.com/validmind/developer-framework/pull/271)--->
<!---[SC-2411] Clustering models support by @AnilSorathiya in [#271](https://github.com/validmind/validmind-library/pull/271)--->
#### Support for clustering models

The {{< var validmind.developer >}} has added support for clustering models. The updates include:
Expand All @@ -33,7 +33,7 @@ The {{< var validmind.developer >}} has added support for clustering models. The

<!---[Try it ...](/notebooks/code_samples/clustering/quickstart_custer_demo.ipynb)--->

<!---John6797/sc 2416/embeddings models support by @johnwalz97 in [#272](https://github.com/validmind/developer-framework/pull/272)--->
<!---John6797/sc 2416/embeddings models support by @johnwalz97 in [#272](https://github.com/validmind/validmind-library/pull/272)--->
#### Support for embeddings models

We added initial support for text embeddings models in the {{< var validmind.developer >}} which enables you to create, use and test a BERT embeddings model utilizing the Hugging Face library. The updates include:
Expand Down
16 changes: 8 additions & 8 deletions site/releases/2023/2023-oct-25/highlights.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ We've introduced new features to the {{< var vm.platform >}} that enable you to
<!-- ### {{< var validmind.developer >}} (v1.22.0) -->

<!--- NR the next three notebooks have been commented out as they are not ready to be released.--->
<!---[SC-2284] Support regression models by @AnilSorathiya in [#259](https://github.com/validmind/developer-framework/pull/259)
<!---[SC-2284] Support regression models by @AnilSorathiya in [#259](https://github.com/validmind/validmind-library/pull/259)
#### Support for regression models

The {{< var validmind.developer >}} has added support for regression models. The updates include:
Expand All @@ -24,7 +24,7 @@ The {{< var validmind.developer >}} has added support for regression models. The

[Try it ...](/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb)--->

<!---[SC-2411] Clustering models support by @AnilSorathiya in [#271](https://github.com/validmind/developer-framework/pull/271)
<!---[SC-2411] Clustering models support by @AnilSorathiya in [#271](https://github.com/validmind/validmind-library/pull/271)
#### Support for clustering models

The {{< var validmind.developer >}} has added support for clustering models. The updates include:
Expand All @@ -35,7 +35,7 @@ The {{< var validmind.developer >}} has added support for clustering models. The

[Try it ...](/notebooks/code_samples/clustering/quickstart_custer_demo.ipynb)--->

<!---John6797/sc 2416/embeddings models support by @johnwalz97 in [#272](https://github.com/validmind/developer-framework/pull/272)
<!---John6797/sc 2416/embeddings models support by @johnwalz97 in [#272](https://github.com/validmind/validmind-library/pull/272)
#### Support for embeddings models

We added initial support for text embeddings models in the {{< var validmind.developer >}} which enables you to create, use and test a BERT embeddings model utilizing the Hugging Face library. The updates include:
Expand All @@ -47,7 +47,7 @@ We added initial support for text embeddings models in the {{< var validmind.dev

[Try it ...](/notebooks/POC/bert-embeddings-model-ow-poc.ipynb)--->

<!---[SC-2236] Demo notebook for changing config/parameters by @AnilSorathiya in [#251](https://github.com/validmind/developer-framework/pull/251)--->
<!---[SC-2236] Demo notebook for changing config/parameters by @AnilSorathiya in [#251](https://github.com/validmind/validmind-library/pull/251)--->
<!-- #### New notebook to demonstrate how to change configuration parameters

This notebook serves as a guide for modifying configuration and parameters within the {{< var validmind.developer >}}. It includes the following features:
Expand Down Expand Up @@ -115,7 +115,7 @@ From this page you can manage:

## Enhancements

<!---[SC-2346] Rouge and Bert score tests should show average scores by @juanmleng in [#263](https://github.com/validmind/developer-framework/pull/263)--->
<!---[SC-2346] Rouge and Bert score tests should show average scores by @juanmleng in [#263](https://github.com/validmind/validmind-library/pull/263)--->
#### Rouge and Bert Score tests now show average scores

:::: {.flex .flex-wrap .justify-around}
Expand All @@ -138,7 +138,7 @@ From this page you can manage:
<!--- NR this notebook is not currently included in our docs site:
Tested these metrics running `foundational_models_summarization_high_code.ipynb` --->

<!---[SC-2143] Tests for safety toxicity and bias in text summarization by @juanmleng in [#258](https://github.com/validmind/developer-framework/pull/258)--->
<!---[SC-2143] Tests for safety toxicity and bias in text summarization by @juanmleng in [#258](https://github.com/validmind/validmind-library/pull/258)--->
#### Tests for safety toxicity and bias in text summarization

We introduced several new tests to evaluate safety and bias risks in text summarization:
Expand Down Expand Up @@ -168,7 +168,7 @@ To test these metrics, see notebook `foundation_models_summarization_bias.ipynb`

## Bug fixes

<!---[SC-2303] Shap test issue resolved by @juanmleng in [#262](https://github.com/validmind/developer-framework/pull/262)--->
<!---[SC-2303] Shap test issue resolved by @juanmleng in [#262](https://github.com/validmind/validmind-library/pull/262)--->
#### Shap test issue resolved
- We set `matplotlib` to version `3.7.x` in `pyproject.toml` to fix an incompatibility with the latest `matplotlib` version (`3.8.0`).
- This incompatibility was causing SHAP plot errors. We will keep track of `matplotlib` releases for future updates. Once fixed, we will consider updating the version.
Expand Down Expand Up @@ -209,7 +209,7 @@ We updated the QuickStart section of our documentation to reflect recent {{< var



<!---John6797/sc 2211/update test descriptions by @johnwalz97 in [#244](https://github.com/validmind/developer-framework/pull/244)
<!---John6797/sc 2211/update test descriptions by @johnwalz97 in [#244](https://github.com/validmind/validmind-library/pull/244)
- **John6797/sc 2211/update test descriptions**. Add full markdown descriptions to all tests
--->

Expand Down
4 changes: 2 additions & 2 deletions site/releases/2024-aug-13/release-notes.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ We enhanced the {{< var validmind.platform >}} **{{< fa home >}} Dashboard** wit

<!---
PR #144: Generalize support for comparison tests
URL: https://github.com/validmind/developer-framework/pull/144
URL: https://github.com/validmind/validmind-library/pull/144
Labels: enhancement
--->
### Generalized support for comparison tests
Expand Down Expand Up @@ -195,7 +195,7 @@ Manage both upstream and downstream model interdependencies:

<!---
PR #140: [SC-5466] Exclude categorical and binary features from outlier tests
URL: https://github.com/validmind/developer-framework/pull/140
URL: https://github.com/validmind/validmind-library/pull/140
Labels: bug
--->
### Exclude categorical & binary features from outlier tests
Expand Down
12 changes: 6 additions & 6 deletions site/releases/2024-feb-14/highlights.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ We've improved the {{< var vm.product >}} user experience, from more supportive

### {{< var validmind.developer >}} (v1.26.6)

<!---[SC-2706] Documentation templates should allow rendering more than one unique test result for the same test by @AnilSorathiya in [#322](https://github.com/validmind/developer-framework/pull/322)--->
<!---[SC-2706] Documentation templates should allow rendering more than one unique test result for the same test by @AnilSorathiya in [#322](https://github.com/validmind/validmind-library/pull/322)--->
#### Support for tracking each test result with a unique identifier

:::: {.flex .flex-wrap .justify-around}
Expand Down Expand Up @@ -73,7 +73,7 @@ test.log()
::::


<!---[SC 3073] `run_documentation_tests()` should allow specifying inputs for any test individually by @AnilSorathiya in [#327](https://github.com/validmind/developer-framework/pull/327)--->
<!---[SC 3073] `run_documentation_tests()` should allow specifying inputs for any test individually by @AnilSorathiya in [#327](https://github.com/validmind/validmind-library/pull/327)--->
#### Easier specification of inputs for individual tests

:::: {.flex .flex-wrap .justify-around}
Expand Down Expand Up @@ -132,7 +132,7 @@ Here, the top-level `inputs` parameter acts as a global `inputs` parameter, and
::::


<!---[SC-2330] Ability to see available task types and tags by @juanmleng in [#317](https://github.com/validmind/developer-framework/pull/317)--->
<!---[SC-2330] Ability to see available task types and tags by @juanmleng in [#317](https://github.com/validmind/validmind-library/pull/317)--->
#### View available task types and tags to filter tests

To enable model developers to know what task types and tags are available to filter on, we have made some updates to our {{< var vm.developer >}}:
Expand All @@ -156,7 +156,7 @@ To enable model developers to know what task types and tags are available to fil

![Explore tests notebook with the Understanding Tags and Task Types code cells run successfully](list-tasks-and-tags.png){width=80% fig-alt="Screenshot showing the Explore tests notebook with the Understanding Tags and Task Types code cells run successfully" width=90% .screenshot}

<!---John6797/sc 2943/ValidMind Library should register documentation by @johnwalz97 in [#324](https://github.com/validmind/developer-framework/pull/324)--->
<!---John6797/sc 2943/ValidMind Library should register documentation by @johnwalz97 in [#324](https://github.com/validmind/validmind-library/pull/324)--->
#### {{< var validmind.developer >}} documentation inputs tracking

:::: {.flex .flex-wrap .justify-around}
Expand Down Expand Up @@ -307,7 +307,7 @@ We implemented a toggle feature in the **Model Activity** and Recent Activity se

## Bug fixes

<!---Invalid content blocks create errors in run documentation by @cachafla in [#326](https://github.com/validmind/developer-framework/pull/326)--->
<!---Invalid content blocks create errors in run documentation by @cachafla in [#326](https://github.com/validmind/validmind-library/pull/326)--->
#### Invalid content blocks for `run_documentation_tests()`

:::: {.flex .flex-wrap .justify-around}
Expand Down Expand Up @@ -396,7 +396,7 @@ We implemented a toggle feature in the **Model Activity** and Recent Activity se

::::

<!---Add input_id usage to notebook samples by @cachafla in [#332](https://github.com/validmind/developer-framework/pull/332)--->
<!---Add input_id usage to notebook samples by @cachafla in [#332](https://github.com/validmind/validmind-library/pull/332)--->
#### Updated sample notebooks with current `input_id` usage
:::: {.flex .flex-wrap .justify-around}

Expand Down
Loading