Revisit the "Developer Framework" getting started section (with bonus tests section rework)

[nrichers]

Internal Notes for Reviewers

This PR includes:

• Get started with the ValidMind Developer Framework — reworked to step new users through the learning process hands on. Gone is the glut of overview or conceptual information you received upfront, replaced with notebooks (and a future video) that you can try out as you form your mental model of the developer framework.
• Run tests and test suites — now groups the how-to notebooks into sections for getting started (beginners), exploring tests & suites, intermediate, and advanced users. This new structure replaces the single 'grab bag of notebooks' where you had to figure out the order of things yourself.

These changes are part of a larger effort to improve the learning journey. There is a follow-on companion PR in the works for the developer-framework repo that will:

• Re-home some conceptual info — information that is relevant to new users needs to be dispersed into the notebooks themselves where it can be provided in the context where you need to know about it.
• Edit notebooks for consistency — the improved emphasis on hands-on notebooks with content tiles you step through highlights the fact that our notebooks often still look and feel disparate.

Output

Get started with the ValidMind Developer Framework

Before	After

Run tests and test suites

Before	After

External Release Notes

The developer documentation has received some improvements to help you get started more easily:

• Get started with the ValidMind Developer Framework: The main page for getting started now guides you through the learning process hands-on with the help of our sample notebooks. Learn more ...
• Run tests and test suites: The available how-to notebooks have been restructured according to experience levels with the developer framework to simplify the learning path. There is also a new interactive test sandbox you can explore. Learn more ...

[validbeck]

Get started with the ValidMind Developer Framework

• The abstract is still a bit clunky, IMO. In particular, I don't feel that it needs to stress "who" uses it (again, thinking of "stories" rather than "personas") since it is already in the feature name. ;) I think it also needs a bit more "value" pep. Perhaps:

The ValidMind Developer Framework helps you streamline model documentation by automating the generation of drafts. All you need to do is upload your documentation artifacts and test results to the ValidMind platform.

Key Validmind concepts

• I like the information here but I think this should actually live in its own Glossary page under the Developer Framework in the sidebar. This can help us establish things like official product {variable} names as well. This way we can make the "The Developer Framework is designed [...]" into a call-out box instead as I think it's important to bring attention to the model-agnostic quirk in particular.

Getting started

• Minor phrasing quibbles here for your consideration:

Quickstart > quickstart guide
video > video demonstration

• Maybe change the header for the Quickstart to something less wordy as well since the description sort of covers the bases. Again I'm not fond of the "developer-only" mindset (something I'm trying to wrap into our style guide, which is "curious-friendly/role-agnostic language"... but I digress), so perhaps:

Model Documentation Quickstart: This interactive notebook guides you through the process of documenting a model with the ValidMind Developer Framework. It uses the Bank Customer Churn Prediction sample dataset from Kaggle (aside: can we link out to this model or its context here? I think that'd be helpful for total beginners to understand what we're working with) to train a simple classification model.

• Do we want to capitalise developer framework under the video description?

Learn how to run tests

• Minor wording suggestion (sounds a bit more personally enticing/agnostic perhaps):

ValidMind provides a variety of built-in tests and test suites which make it easy for you to automate your model documentation.

• Running an Individual Test: Is this meant to have nothing after the required to: opener?
• Also, are the tests actually proper nouns/established product names or can we go with sentence case convention?
• Configure Parameters for a Specific Test:

This notebook guides you through using a simple [...]

• Ah, I see, it captures the abstract from the actual pages itself. We may want to modify them so they make sense on the CTA cards.

Try the code samples

• Again a preference for just third person here instead of the role, which is something I stole from your draft of our style guide!

What's next

• Not sure if developer framework here is meant to be capitalised as a proper noun (yyyyes...?)...
• Can we maybe turn some of these into links? (If we have the relevant documentation!)

Other thoughts

• I love the new layout! Much more approachable. :)
• We may want to insert a call-out to the glossary under the CTA cards in Getting started if we end up deciding on the new page

---

Run tests and test suites

• More nitpicking at the abstract as this one is quite run-on and wordy:

ValidMind provides a variety of built-in tests and test suites which help you produce documentation during stages of the model development lifecycle where you need to validate that your work satisfies MRM (model risk management) requirements.

Key concepts

• Again I think we should consolidate these concepts that many pages refer to in a glossary that can be linked out to instead of having components that need to be updated individual on each page, thoughts?

When do I use tests and tests suites?

• I would just remove the As a model developer, opening here.

Can I use my own tests?

• Yes > "Absolutely!" (I want our excitement to be contagious, dangit.)

Reference

• Good opportunity to link to the glossary perhaps?

Other thoughts

• I'll take a look at the introductory blurbs for all these CTA cards in another comment. Stay tuned!

[validbeck]

CTA abstract edits

• As promised!

Quickstart for Customer Churn Model Documentation — Full Suite

• Title shortened as above comment
• Ah, I see we do link out to the context for the dataset, excellent

Running an Individual Test

• I think we can just put a line-break here between the first and second sentence so that the CTA card doesn't retrieve an empty colon opener.

Configure Parameters for a Specific Test

• Kaggle link here is plaintext instead of rendering as a proper hyperlink and there are quite a few grammatical errors/comprehension dead-ends as well
• Here's my suggestion for everything that should go in the first paragraph so it's captured in the CTA:

This notebook guides you through using a simple classification model for a bank customer churn dataset. It shows you how to set up the ValidMInd Developer Framework, and guides you through documenting a model within the framework.

• Everything else can be part of a second paragraph, methinks:

You will learn how you can configure parameters for a test or set of tests in a specific section of your document. For this simple demonstration, we will use a bank customer churn dataset from [Kaggle].

Implementing Custom Metrics and Threshold Tests

• First paragraph:

Custom metrics offer added flexibility by extending the defaults provided by ValidMind, enabling you to document any type of model or use case.

• Second paragraph:

Both metrics and threshold tests assess models but they differ in approach: [...]

• I would break the These instructions [...] into a fresh line as well

Customizing Metric Outputs using Output Templates

• 1st:

This notebook guides you through customizing the standard outputs produced by running a suite of default tests with the ValidMind Developer Framework.

• 2nd: It uses the [...] + As part of the notebook [...]

Prompt Validation for Large Language Models (LLMs)

• 1st:

This notebook guides you through using ValidMind for running and documenting prompt validation tests for a large language model (LLM) specialized in sentiment analysis for financial news. It shows you how to set up the ValidMind Developer Framework, initialize the client library, and use a specific prompt template for analyzing the sentiment of given sentences.

• 2nd:

The prompt validation covers the initialization of a test dataset and the creation of a foundational model using ValidMind’s framework, followed by the execution of a test suite specifically designed for prompt validation. The notebook also includes example data to test the model’s ability to correctly identify sentiment as positive, negative, or neutral.

Time Series Forecasting Model Tutorial

• Some run-ons and grammar oopsies in this one.
• 1st:

This notebook guides you through the process of automatically documenting and testing time series forecasting models using ValidMind.

• 2nd:

You will use the ValidMind Developer Framework to import and prepare data, before running a data validation test suite, followed by loading a pre-trained model. Finally, you will run a model validation test suite!

Run tests and test suites

• See comment above.

[nrichers]

@validbeck this is awesome feedback, thank you! There's a few follow-up items I've jotted down where we should chat further (key terms into the glossary, for example, or style guide items). We can chat about these when we meet next. Also, the CTA edits you provided are most welcome but I will need to address them in a PR for the developer-framework repo — not a problem as I'm currently editing those notebooks already.

Have you tried the suggestion feature in GitHub? You can make suggested edits as comments that can be committed directly. It avoids having to transcribe edits, which for me at least is often error-prone, and your suggestions count towards your GitHub contributions (commits include a co-authored-by line that credits you).

[validbeck]

Have you tried the suggestion feature in GitHub? You can make suggested edits as comments that can be committed directly. It avoids having to transcribe edits, which for me at least is often error-prone, and your suggestions count towards your GitHub contributions (commits include a co-authored-by line that credits you).

I have now tried this feature out! Hopefully I did it correctly. :)

[validbeck]

Beep boop, I added some suggestions and I hope they show up!

[validbeck]

Now with actual suggestions!