Release notes to Sprint 66 (25.01)

[nrichers]

Internal Notes for Reviewers

This PR includes the release notes to Sprint 66, along with refreshed notebooks, Python API reference, and test descriptions.

• Preview: January 31, 2025
• Source: release-notes.qmd

Notes

• This is the first release to follow our new versioning scheme (25.01) — I added a note at the top to that effect.
• All admin UI PRs are commented out for lack of detail, except Configurable default models — should this be there?

Other improvements

• @validbeck: PR previews were broken due to demo notebook breaking changes. You can now toggle notebook execution using ENABLE_DEMO_NOTEBOOK.

External Release Notes

[github-actions]

PR Summary

This pull request introduces several enhancements and bugfixes to the ValidMind software project, focusing on documentation, monitoring, and testing capabilities. Key changes include:

1. Conditional Execution in GitHub Actions: The demo notebook execution in the GitHub workflow is now conditional on the ENABLE_DEMO_NOTEBOOK environment variable being set to true.

2. Documentation Updates: The release notes for January 31, 2025, have been added, and the site index has been updated to include the latest release notes.

3. New Notebooks and Tests: Several new Jupyter notebooks and tests have been introduced, focusing on credit risk scorecard models and ongoing monitoring capabilities. These include:

   • Application scorecard model documentation using XGBoost.
   • Ongoing monitoring for application scorecard models.
   • Custom tests for score band discrimination metrics.

4. Code Quality Improvements: Refactoring of code samples to remove unnecessary conditional checks and improve readability.

5. New Features: Introduction of custom dashboards, LLM-based description context, and logging metrics over time with threshold visualization.

6. Security and Compliance: Enhancements to ensure secure handling of model and dataset initialization, as well as improved documentation for compliance with regulatory standards.

Test Suggestions

• Test the conditional execution of the demo notebook in the GitHub workflow by setting the ENABLE_DEMO_NOTEBOOK environment variable to both true and false.
• Verify the inclusion of the latest release notes in the site index and ensure they are correctly displayed.
• Run the new Jupyter notebooks to ensure they execute without errors and produce the expected documentation and monitoring outputs.
• Test the new custom tests for score band discrimination metrics to ensure they provide accurate and meaningful results.
• Validate the functionality of the new custom dashboards and LLM-based description context features.
• Check the logging of metrics over time with threshold visualization to ensure thresholds are correctly displayed and updated.

[github-actions]

PR Summary

This pull request introduces several enhancements and bugfixes to the ValidMind software project, focusing on documentation, monitoring, and model validation capabilities. Key changes include:

1. Workflow Enhancements:

   • Updated the GitHub workflow to conditionally execute the demo notebook based on the ENABLE_DEMO_NOTEBOOK environment variable.

2. Documentation Updates:

   • Added a new release note for January 31, 2025, detailing the latest features and improvements.
   • Updated the site index and release notes to include the latest release information.

3. New Notebooks and Code Samples:

   • Introduced new Jupyter notebooks for documenting and monitoring application scorecard models using the ValidMind Library.
   • Added a notebook for ongoing monitoring of application scorecard models, showcasing how to track and visualize model performance metrics over time.
   • Provided a notebook for logging metrics over time, demonstrating how to track key model performance metrics such as AUC, F1 score, precision, recall, and accuracy.

4. Custom Tests and Enhancements:

   • Added a custom test for evaluating score band discrimination metrics in credit risk assessment.
   • Introduced several new tests for ongoing monitoring, including calibration curve drift, class discrimination drift, and feature drift.

5. Code Quality Improvements:

   • Simplified code in various notebooks by removing unnecessary conditional checks and improving readability.
   • Updated test descriptions to remove LaTeX notation and provide clearer explanations.

6. Security and Compliance:

   • Ensured that all new features and tests comply with security and data privacy standards.

These changes aim to enhance the functionality, usability, and security of the ValidMind platform, providing users with more robust tools for model documentation and monitoring.

Test Suggestions

• Test the conditional execution of the demo notebook by setting the ENABLE_DEMO_NOTEBOOK environment variable to true and false.
• Verify that the new release notes are correctly displayed on the site.
• Run the new Jupyter notebooks to ensure they execute without errors and produce the expected outputs.
• Test the custom score band discrimination metrics to ensure they provide accurate results.
• Validate the new ongoing monitoring tests for calibration curve drift, class discrimination drift, and feature drift.
• Check the updated test descriptions for clarity and accuracy.

[github-actions]

A PR preview is available: Preview URL

[validbeck]

PR previews were broken due to demo notebook breaking changes. You can now toggle notebook execution using ENABLE_DEMO_NOTEBOOK.

Neat! Will be fixed in PR #633 and the issue was reviewed and resolved by John but good to have in the future.

[validbeck]

Left some comments/suggestions inline for improvements. Also:

The images in the release folder need to have human-readable names, please. Half the audit effort was me renaming files called hskjfhjskfg.png etc. I know this is a minor quibble but this is just a good habit to have, especially as it is outlined in our style guide.

EDIT: Also, the contents of the "Release highlights" section should be encased in the .highlights class and use the green buttons:

Especially for long releases like this this is a convention we introduced to call attention to the more important bits.

[nrichers]

EDIT: Also, the contents of the "Release highlights" section should be encased in the .highlights class and use the green buttons:

I feel like this is another area where we've added a layer of complexity to the reading and authoring experience that might best be pruned again? I don't think this buttonology adds anything of value to the reader, and it makes the authoring experience worse.

From what I recall ...

• we started adding buttons for highlights to make them stand out more;
• next, buttons crept into many of the links throughout the release notes; I objected but then relented
• and now we are differentiating between buttons for highlights and other release note items that have to be coded differently

The authoring experience should be able to focus on the content — the fact that one button is solid dark green and the other a pink outline is a nuance that doesn't really add to the storytelling about the release.

Your other review comments all make sense and I will address them. That said, I have a concern here about how much time we put into release notes — all of the spit & polish items you suggested make sense, but they take time.

[mehdi0501]

Left some comments throughout. There were a couple of new features/enhancements that were in parallel of sprint 66, that I could not find in the notes (check with @cachafla ), but which warrant a call out. Apologies if we talk about them elsewhere

• the enhanced metadata descriptions in the UI
• new math equation editor and support for LateX formulas
• export model inventory
• export findings

[nrichers]

The images in the release folder need to have human-readable names, please. Half the audit effort was me renaming files called hskjfhjskfg.png etc. I know this is a minor quibble but this is just a good habit to have, especially as it is outlined in our style guide.

Renamed as suggested (and removed some images we're not actually using).

[github-actions]

A PR preview is available: Preview URL

[validbeck]

A note as I edit: releases need a summary at the top for the CTA. Please try to do these as we write releases so the person cleaning up yearly releases doesn't have to? Also, since we surface these on our index landing page, it needs the summary to incite people to read.

[github-actions]

A PR preview is available: Preview URL

[github-actions]

A PR preview is available: Preview URL

[validbeck]

In the interest of time I've cleaned things up and made the edits, but I did catch a few broken links/some .qmd links out to files that weren't the Python docs as an FYI.

[github-actions]

A PR preview is available: Preview URL

[nrichers]

@mehdi0501:

• the enhanced metadata descriptions in the UI

Still investigating ...

• new math equation editor and support for LateX formulas

This was a fix in Sprint 66 but the related frontend PR was marked as internal, meaning we skipped it for the release notes. Docs were added retroactively via #631

-> Added to the release notes via 35b9dc5

• https://app.shortcut.com/validmind/story/7976/add-ckeditor5-math-support-deprecate-mathtype
• https://github.com/validmind/frontend/pull/1156

• export model inventory
• export findings

Both of these were added in Sprint 64 but the related frontend PR was marked as internal, skipped for the release notes. It doesn't look like we ever flagged the related story as needing documentation, either.

-> tracked via sc-7539

• https://app.shortcut.com/validmind/story/7539/initial-support-for-inventory-and-findings-export
• https://github.com/validmind/frontend/pull/1107

@cachafla looks like there's a couple of process breakdowns where

• user guide updates would have been required but were not flagged
• PRs that should have had release notes were labelled as internal

[github-actions]

A PR preview is available: Preview URL

[github-actions]

A PR preview is available: Preview URL

[github-actions]

A PR preview is available: Preview URL

[nrichers]

In the interest of time I've cleaned things up and made the edits, but I did catch a few broken links/some .qmd links out to files that weren't the Python docs as an FYI.

Thank you for all your editing help, @validbeck! I found a couple more links that needed fixing, but I think these release notes look very clean now. I especially like how you moved some content into the margin.

(I like to gripe about how much time we spend on release notes but, when you do, the results speak for themselves.)

[github-actions]

A PR preview is available: Preview URL

[github-actions]

A PR preview is available: Preview URL