Generate Quarto Markdown for Python API

[nrichers]

Internal Notes for Reviewers

This PR generates Quarto Markdown documentation for our Python API with make quarto-docs. These Quarto docs can then be fetched and rendered with the main docs site: LIVE PREVIEW

How make quarto-docs works:

1. Griffe extracts API information from the API codebase and dumps it to a JSON file
2. A Python script extracts the API information from the JSON, processes it, and passes it to Jinja2 templates
3. Jinja2 templates and shared macros transform this information into Quarto Markdown files
4. The script generates a sidebar navigation fragment based on the output file structure
5. CI/CD integration tests the Quarto docs generation and commits the output

Relates to: validmind/documentation#640

What is nice about this PR

• Integrates the Python API docs with our main site — Looks and feels like ValidMind developer docs, enables one search for the entire site, and enables other cool stuff in the future
• Gives us more control — Modular templates and macros provide us with fine-grained control over how we want content to appear^*
• Doesn't touch the pdoc stuff — Current HTML docs generation continues as before, with a new workflow to also generate Quarto docs
• Fixes some API codebase issues — While troubleshooting, I fixed some stuff, like a missing __init__.py file, missing type annotations, and docstrings.

^* Examples:

• Instead of just __version__, we output the actual version as a link in the sidebar (see the screenshot above)
• With full control of the navigation, we can display more content without pdoc's constantly swapping the entire sidebar
• Error messages are now grouped (models, tests, etc.) instead of being in a single large list
• Codeblocks use ValidMind colors, allow links, and provide clean formatting for signatures
• Init root functions come first for a cleaner flow (some improvements are in the API codebase and will also benefit pdoc HTML output)

Summary of the changes

Refer to README.md for how Quarto Markdown generation works.

Makefile

• make quarto-docs — Added new action to generate Quarto Markdown from the Python API

.github/

• integration.yaml and python.yaml — Updated workflows to test Quarto Markdown generation and ignore all of docs/ (not just docs/_build/)
• quarto-docs.yaml — Added new workflow to generate Quarto docs and commit them

docs/

• README.md — Added to explain the Quarto Markdown generation process
• templates/ — Added Quarto templates and macros based on Jinja2
• validmind.css — Added CSS styling for Quarto Markdown
• _metadata.yml — Added Quarto configuration file
• _sidebar.yml — Generated Quarto sidebar navigation fragment
• validmind.qmd, validmind/ — Generated Quarto Markdown files for the ValidMind Python API

scripts/

• generate_quarto_docs.py — Added documentation-generating script that:
  • Extracts API information from the codebase using the JSON dump from Griffe
  • Passes JSON to Jinja2 templates and macros to generate Quarto files and sidebar navigation
  • Lints and writes output to docs/

Features

• Private/public filtering — Filtering of which members to include in documentation, respecting all lists
• Root module handling — Processing for the main validmind module, including version information extraction
• Alias resolution — Resolves imported symbols to their original definitions, with caching to improve performance
• Docstring normalization — Pre-processes docstrings to fix formatting issues, particularly for parameter descriptions
• Class finding across modules — Recursively searches for classes when they can't be found through direct imports
• Inherited member handling — Handling for documenting inherited methods
• Errors module handling — Dedicated template with custom sorting that prioritizes base errors first, then groups by error category
• Test suites handling — Logic for test suite classes that may be aliased across modules
• VM models special case handling — Ensures ResultTable and TestResult classes are properly documented
• Specific exclusions — Explicitly excludes internal utilities like SkipTestError and logger from test modules
• Sidebar generation — Create a hierarchical navigation structure from the flat file list for Quarto

validmind/

• __init__.py — Reordered __all__ list for better content flow in the public API
• vm_models/test_suite/__init__.py — Added new init file to properly expose module structure
• Docstrings — Edited across a number of modules to provide more detailed explanations, parameter descriptions, and return type annotations
• Type annotations — Fixed and standardized throughout to ensure consistency and correctness, particularly for return types

Other changes

• .gitignore — Updated to ignore Quarto-specific files we don't want to commit
• pyproject.toml — Updated with new dependencies for documentation generation

Limitations

Function signatures use HTML elements to work around the mdformat linter removing nested [] brackets even with escaping which breaks some output. You could turn off linting but this then breaks parameter descriptions that do need to escape some Markdown characters. There are ways around this issue in the future.

For future reference:

• Why does mdformat backslash escape special syntax specific to MkDocs / Hugo / Obsidian / GitHub / some other Markdown engine?

Notes

Fixes to the Python API — PLEASE CHECK CAREFULLY!

There were quite a few times when I hit issues that turned out to originate in our Python API itself.

Simple docstrings fixes

These fixes are generally safe:

• Continuation line indenting
• Stray RST-style docstrings
• Minor text edits, mainly punctuation and sentence structure

Missing validmind/vm_models/test_suite/__init__.py

I spent about 6 hours trying to figure out why I wasn't getting part of the API in the JSON dump and eventually traced it to a missing init file in a folder, added via c549bee.

(pdoc seems to be quite permissive but Griffe will not inspect this part of the API without an init file.)

JSON dump warnings

These commits fixed the following warnings when dumping the JSON from the API with Griffe:

• 1a72190
• 55a7e15
• c835224

Before	After

For example, we now correctly indicate more optional parameters and return annotations:

Before	After

Before	After

Output file comparison

The Quarto Markdown files are a drop-in replacements with only minor differences:

• pdoc HTML: Includes search stuff, handled by Quarto
• Quarto Markdown: Includes generated sidebar fragment, metadata, and CSS files
• Quarto HTML: Adds index.html alias to match the pdoc

pdoc HTML	Quarto Markdown	Quarto HTML

Minor differences, FYI

Discrepancies between signatures and docstrings

Here, we render what is in the JSON for the function signature which happens to match the docstring better:

pdoc	Quarto

Here, we directly handle the callable kind from the JSON metadata while preserving Python's type hint syntax, unlike the old HTML which fails to resolve the type correctly with a mangled NoneType. Also, we don't repeat the link as it's already provided by the return annotation:

pdoc	Quarto

Here, we're using a common convention when using Plotly — importing the graph_objs module as go to make the function signature more concise (note how pdoc runs out of space). Both versions refer to the same underlying Plotly types, just using different import styles:

pdoc	Quarto

External Release Notes

[nrichers]

I found a fix for a quirk we inherited from our pdoc HTML reference. The issue is that we get a very L-O-N-G list of individual test IDs with no links at all — in the Quarto docs, we just list and link to the test category pages which then list the indidividual tests.

pdoc	Quarto

Just how long is that list? Compare this snippet in the signature:

pdoc

Quarto

test_id: Union[Literal['validmind.data_validation.ACFandPACFPlot', 'validmind.data_validation.ADF', 'validmind.data_validation.AutoAR', 'validmind.data_validation.AutoMA', 'validmind.data_validation.AutoStationarity', 'validmind.data_validation.BivariateScatterPlots', 'validmind.data_validation.BoxPierce', 'validmind.data_validation.ChiSquaredFeaturesTable', 'validmind.data_validation.ClassImbalance', 'validmind.data_validation.DatasetDescription', 'validmind.data_validation.DatasetSplit', 'validmind.data_validation.DescriptiveStatistics', 'validmind.data_validation.DickeyFullerGLS', 'validmind.data_validation.Duplicates', 'validmind.data_validation.EngleGrangerCoint', 'validmind.data_validation.FeatureTargetCorrelationPlot', 'validmind.data_validation.HighCardinality', 'validmind.data_validation.HighPearsonCorrelation', 'validmind.data_validation.IQROutliersBarPlot', 'validmind.data_validation.IQROutliersTable', 'validmind.data_validation.IsolationForestOutliers', 'validmind.data_validation.JarqueBera', 'validmind.data_validation.KPSS', 'validmind.data_validation.LJungBox', 'validmind.data_validation.LaggedCorrelationHeatmap', 'validmind.data_validation.MissingValues', 'validmind.data_validation.MissingValuesBarPlot', 'validmind.data_validation.MutualInformation', 'validmind.data_validation.PearsonCorrelationMatrix', 'validmind.data_validation.PhillipsPerronArch', 'validmind.data_validation.ProtectedClassesCombination', 'validmind.data_validation.ProtectedClassesDescription', 'validmind.data_validation.ProtectedClassesDisparity', 'validmind.data_validation.ProtectedClassesThresholdOptimizer', 'validmind.data_validation.RollingStatsPlot', 'validmind.data_validation.RunsTest', 'validmind.data_validation.ScatterPlot', 'validmind.data_validation.ScoreBandDefaultRates', 'validmind.data_validation.SeasonalDecompose', 'validmind.data_validation.ShapiroWilk', 'validmind.data_validation.Skewness', 'validmind.data_validation.SpreadPlot', 'validmind.data_validation.TabularCategoricalBarPlots', 'validmind.data_validation.TabularDateTimeHistograms', 'validmind.data_validation.TabularDescriptionTables', 'validmind.data_validation.TabularNumericalHistograms', 'validmind.data_validation.TargetRateBarPlots', 'validmind.data_validation.TimeSeriesDescription', 'validmind.data_validation.TimeSeriesDescriptiveStatistics', 'validmind.data_validation.TimeSeriesFrequency', 'validmind.data_validation.TimeSeriesHistogram', 'validmind.data_validation.TimeSeriesLinePlot', 'validmind.data_validation.TimeSeriesMissingValues', 'validmind.data_validation.TimeSeriesOutliers', 'validmind.data_validation.TooManyZeroValues', 'validmind.data_validation.UniqueRows', 'validmind.data_validation.WOEBinPlots', 'validmind.data_validation.WOEBinTable', 'validmind.data_validation.ZivotAndrewsArch', 'validmind.data_validation.nlp.CommonWords', 'validmind.data_validation.nlp.Hashtags', 'validmind.data_validation.nlp.LanguageDetection', 'validmind.data_validation.nlp.Mentions', 'validmind.data_validation.nlp.PolarityAndSubjectivity', 'validmind.data_validation.nlp.Punctuations', 'validmind.data_validation.nlp.Sentiment', 'validmind.data_validation.nlp.StopWords', 'validmind.data_validation.nlp.TextDescription', 'validmind.data_validation.nlp.Toxicity', 'validmind.model_validation.BertScore', 'validmind.model_validation.BleuScore', 'validmind.model_validation.ClusterSizeDistribution', 'validmind.model_validation.ContextualRecall', 'validmind.model_validation.FeaturesAUC', 'validmind.model_validation.MeteorScore', 'validmind.model_validation.ModelMetadata', 'validmind.model_validation.ModelPredictionResiduals', 'validmind.model_validation.RegardScore', 'validmind.model_validation.RegressionResidualsPlot', 'validmind.model_validation.RougeScore', 'validmind.model_validation.TimeSeriesPredictionWithCI', 'validmind.model_validation.TimeSeriesPredictionsPlot', 'validmind.model_validation.TimeSeriesR2SquareBySegments', 'validmind.model_validation.TokenDisparity', 'validmind.model_validation.ToxicityScore', 'validmind.model_validation.embeddings.ClusterDistribution', 'validmind.model_validation.embeddings.CosineSimilarityComparison', 'validmind.model_validation.embeddings.CosineSimilarityDistribution', 'validmind.model_validation.embeddings.CosineSimilarityHeatmap', 'validmind.model_validation.embeddings.DescriptiveAnalytics', 'validmind.model_validation.embeddings.EmbeddingsVisualization2D', 'validmind.model_validation.embeddings.EuclideanDistanceComparison', 'validmind.model_validation.embeddings.EuclideanDistanceHeatmap', 'validmind.model_validation.embeddings.PCAComponentsPairwisePlots', 'validmind.model_validation.embeddings.StabilityAnalysisKeyword', 'validmind.model_validation.embeddings.StabilityAnalysisRandomNoise', 'validmind.model_validation.embeddings.StabilityAnalysisSynonyms', 'validmind.model_validation.embeddings.StabilityAnalysisTranslation', 'validmind.model_validation.embeddings.TSNEComponentsPairwisePlots', 'validmind.model_validation.ragas.AnswerCorrectness', 'validmind.model_validation.ragas.AspectCritic', 'validmind.model_validation.ragas.ContextEntityRecall', 'validmind.model_validation.ragas.ContextPrecision', 'validmind.model_validation.ragas.ContextPrecisionWithoutReference', 'validmind.model_validation.ragas.ContextRecall', 'validmind.model_validation.ragas.Faithfulness', 'validmind.model_validation.ragas.NoiseSensitivity', 'validmind.model_validation.ragas.ResponseRelevancy', 'validmind.model_validation.ragas.SemanticSimilarity', 'validmind.model_validation.sklearn.AdjustedMutualInformation', 'validmind.model_validation.sklearn.AdjustedRandIndex', 'validmind.model_validation.sklearn.CalibrationCurve', 'validmind.model_validation.sklearn.ClassifierPerformance', 'validmind.model_validation.sklearn.ClassifierThresholdOptimization', 'validmind.model_validation.sklearn.ClusterCosineSimilarity', 'validmind.model_validation.sklearn.ClusterPerformanceMetrics', 'validmind.model_validation.sklearn.CompletenessScore', 'validmind.model_validation.sklearn.ConfusionMatrix', 'validmind.model_validation.sklearn.FeatureImportance', 'validmind.model_validation.sklearn.FowlkesMallowsScore', 'validmind.model_validation.sklearn.HomogeneityScore', 'validmind.model_validation.sklearn.HyperParametersTuning', 'validmind.model_validation.sklearn.KMeansClustersOptimization', 'validmind.model_validation.sklearn.MinimumAccuracy', 'validmind.model_validation.sklearn.MinimumF1Score', 'validmind.model_validation.sklearn.MinimumROCAUCScore', 'validmind.model_validation.sklearn.ModelParameters', 'validmind.model_validation.sklearn.ModelsPerformanceComparison', 'validmind.model_validation.sklearn.OverfitDiagnosis', 'validmind.model_validation.sklearn.PermutationFeatureImportance', 'validmind.model_validation.sklearn.PopulationStabilityIndex', 'validmind.model_validation.sklearn.PrecisionRecallCurve', 'validmind.model_validation.sklearn.ROCCurve', 'validmind.model_validation.sklearn.RegressionErrors', 'validmind.model_validation.sklearn.RegressionErrorsComparison', 'validmind.model_validation.sklearn.RegressionPerformance', 'validmind.model_validation.sklearn.RegressionR2Square', 'validmind.model_validation.sklearn.RegressionR2SquareComparison', 'validmind.model_validation.sklearn.RobustnessDiagnosis', 'validmind.model_validation.sklearn.SHAPGlobalImportance', 'validmind.model_validation.sklearn.ScoreProbabilityAlignment', 'validmind.model_validation.sklearn.SilhouettePlot', 'validmind.model_validation.sklearn.TrainingTestDegradation', 'validmind.model_validation.sklearn.VMeasure', 'validmind.model_validation.sklearn.WeakspotsDiagnosis', 'validmind.model_validation.statsmodels.AutoARIMA', 'validmind.model_validation.statsmodels.CumulativePredictionProbabilities', 'validmind.model_validation.statsmodels.DurbinWatsonTest', 'validmind.model_validation.statsmodels.GINITable', 'validmind.model_validation.statsmodels.KolmogorovSmirnov', 'validmind.model_validation.statsmodels.Lilliefors', 'validmind.model_validation.statsmodels.PredictionProbabilitiesHistogram', 'validmind.model_validation.statsmodels.RegressionCoeffs', 'validmind.model_validation.statsmodels.RegressionFeatureSignificance', 'validmind.model_validation.statsmodels.RegressionModelForecastPlot', 'validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels', 'validmind.model_validation.statsmodels.RegressionModelSensitivityPlot', 'validmind.model_validation.statsmodels.RegressionModelSummary', 'validmind.model_validation.statsmodels.RegressionPermutationFeatureImportance', 'validmind.model_validation.statsmodels.ScorecardHistogram', 'validmind.ongoing_monitoring.CalibrationCurveDrift', 'validmind.ongoing_monitoring.ClassDiscriminationDrift', 'validmind.ongoing_monitoring.ClassImbalanceDrift', 'validmind.ongoing_monitoring.ClassificationAccuracyDrift', 'validmind.ongoing_monitoring.ConfusionMatrixDrift', 'validmind.ongoing_monitoring.CumulativePredictionProbabilitiesDrift', 'validmind.ongoing_monitoring.FeatureDrift', 'validmind.ongoing_monitoring.PredictionAcrossEachFeature', 'validmind.ongoing_monitoring.PredictionCorrelation', 'validmind.ongoing_monitoring.PredictionProbabilitiesHistogramDrift', 'validmind.ongoing_monitoring.PredictionQuantilesAcrossFeatures', 'validmind.ongoing_monitoring.ROCCurveDrift', 'validmind.ongoing_monitoring.ScoreBandsDrift', 'validmind.ongoing_monitoring.ScorecardHistogramDrift', 'validmind.ongoing_monitoring.TargetPredictionDistributionPlot', 'validmind.prompt_validation.Bias', 'validmind.prompt_validation.Clarity', 'validmind.prompt_validation.Conciseness', 'validmind.prompt_validation.Delimitation', 'validmind.prompt_validation.NegativeInstruction', 'validmind.prompt_validation.Robustness', 'validmind.prompt_validation.Specificity', 'validmind.unit_metrics.classification.Accuracy', 'validmind.unit_metrics.classification.F1', 'validmind.unit_metrics.classification.Precision', 'validmind.unit_metrics.classification.ROC_AUC', 'validmind.unit_metrics.classification.Recall', 'validmind.unit_metrics.regression.AdjustedRSquaredScore', 'validmind.unit_metrics.regression.GiniCoefficient', 'validmind.unit_metrics.regression.HuberLoss', 'validmind.unit_metrics.regression.KolmogorovSmirnovStatistic', 'validmind.unit_metrics.regression.MeanAbsoluteError', 'validmind.unit_metrics.regression.MeanAbsolutePercentageError', 'validmind.unit_metrics.regression.MeanBiasDeviation', 'validmind.unit_metrics.regression.MeanSquaredError', 'validmind.unit_metrics.regression.QuantileLoss', 'validmind.unit_metrics.regression.RSquaredScore', 'validmind.unit_metrics.regression.RootMeanSquaredError'], str, NoneType] = None,

test_id:Optional[TestID (Union of validmind.data_validation.*, validmind.model_validation.*, validmind.prompt_validation.* and str)]=None,

Here's one of the pages we now link to:

EDIT

I did the same for unit metrics, though we currently don't generate separate pages for all the unit metrics to link to — not all test IDs are unit metrics but all unit metrics technically are also test IDs:

[cachafla]

Incredible 🫡.

One final teeny-tiny detail: the single quotes are rendered weirdly:

i.e. ’ instead of '.

Other than that it looks really good 👏 amazing!

I would suggest we do a follow up where we try to enforce best practices in terms of naming/typing parameters so documentation stays consistent.

It can start as a "reminder" but I think we can probably somehow add a rule to the linter?

[nrichers]

One final teeny-tiny detail: the single quotes are rendered weirdly

OK, this is finally fixed via 46048af! It turns out Quarto enables Pandoc smart quotes by default but I tracked down the correct fix after some experimentation. We now get the correct ':

[github-actions]

PR Summary

This pull request introduces significant enhancements to the documentation generation process by integrating Quarto for generating API documentation. The changes include:

1. Workflow Updates:

   • The GitHub Actions workflows (integration.yaml and python.yaml) have been updated to ignore all paths under docs/ instead of just docs/_build/. This change ensures that documentation changes do not trigger unnecessary workflow runs.
   • A new job, Generate Quarto Docs, has been added to the python.yaml workflow to build Quarto documentation.

2. New Quarto Documentation Workflow:

   • A new GitHub Actions workflow file, quarto-docs.yaml, has been introduced. This workflow is responsible for generating Quarto documentation using Griffe for API extraction and Jinja2 templates for rendering.
   • The workflow installs necessary dependencies, including Poetry and specific Python packages, to facilitate the documentation generation process.
   • The generated documentation is committed back to the repository, ensuring that the latest documentation is always available.

3. Makefile Enhancements:

   • A new target, quarto-docs, has been added to the Makefile. This target automates the process of cleaning old documentation files, generating API JSON dumps, and creating Quarto documentation using a Python script.

4. Documentation Structure:

   • A comprehensive set of templates and configuration files for Quarto have been added under the docs/ directory. These include Jinja2 templates for various documentation components, CSS for styling, and configuration files for Quarto.
   • The documentation now includes detailed descriptions, visualizations, and structured navigation to improve usability and accessibility.

5. .gitignore Update:

   • The .gitignore file has been updated to exclude docs/validmind.json, which is generated during the documentation process.

These changes collectively enhance the documentation generation process, making it more robust, automated, and aligned with modern documentation practices using Quarto.

Test Suggestions

• Verify that the Quarto documentation is generated correctly by running the make quarto-docs command locally.
• Ensure that the new GitHub Actions workflow quarto-docs.yaml triggers correctly on pushes to the main branch and generates the expected documentation.
• Check that the generated documentation is committed back to the repository as expected.
• Test the quarto-docs target in the Makefile to ensure it performs all steps without errors.
• Validate that the documentation structure and content are correctly rendered and accessible via the generated Quarto site.