[doc] Automate MkDocs Documentation for Fast-LLM Configuration Options

[tscholak]

🧐 Problem Description

Fast-LLM users struggle to understand and navigate the available configuration options. While example configuration files or templates may exist, they lack completeness and clarity about how options should be used, what they mean, or how they interact. This ultimately forces users to consult the configuration code in the Fast-LLM codebase, which is highly abstracted and spread across multiple files. This process is tedious, error-prone, and inaccessible for most users.

Fast-LLM configurations are defined by data classes, where each field includes:

• Natural language descriptions.
• Default values and possible values.
• Validation rules and computed defaults.
• Relationships and dependencies with other fields.

The hierarchical structure of the data classes corresponds to the nested YAML format used for configurations. Understanding this hierarchy is critical for crafting valid configuration files, yet no centralized resource currently exists to help users navigate this complexity.

💡 Proposed Solution

Extend the existing MkDocs documentation pipeline to generate pages for configuration options. This solution should:

1. Parse the Fast-LLM configuration data classes to:
   • Extract field names, default values, possible values, descriptions, and dependencies.
   • Map hierarchical relationships to reflect the YAML structure.
   • Document computed default values and validation rules where possible.
2. Generate structured Markdown pages compatible with MkDocs Material.
3. Integrate with MkDocs hooks so the documentation is automatically updated during the GitHub Action that builds the documentation.

The resulting documentation should allow users to easily browse and search configuration options, including their purpose, default values, and how they interact with other fields.

🔄 Alternatives Considered

• Manual Documentation: Infeasible to maintain and prone to becoming outdated as the codebase evolves.
• Relying on Example Configuration Files: Insufficient for covering the full range of options and their nuances.

📈 Potential Benefits

• Improved Usability: Users can quickly find and understand configuration options without delving into the codebase.
• Better Onboarding: Simplifies the learning curve for new users and contributors.
• Reduced Support Overhead: Minimizes user confusion and questions about configuration details.
• Consistent Maintenance: Ensures the documentation remains accurate as the codebase changes.

📝 Additional Context

• Fast-LLM configurations rely on nested YAML, and the generated documentation must capture this structure accurately.
• Inspiration can be drawn from tools like Hydra and Pydantic, which provide similar mechanisms for documenting structured configurations.