Description
🎯 Goal (What & Why)
Improve Fast-LLM’s validation pipeline by enabling multiple independent validation datasets, allowing better insights into model performance across different domains. This will streamline evaluations, reduce manual steps, and align with best practices in modern LLM training.
🚀 Execution Plan
Step: Implement Multi-Dataset Validation (LM-Loss/Perplexity)
- Modify Fast-LLM’s training config to allow defining multiple validation datasets.
- During validation, compute LM loss (cross-entropy/perplexity) separately for each dataset.
- Ensure results are logged independently in WandB.
📌 Acceptance Criteria (Must-Haves for Completion)
- Multi-dataset LM-loss evaluation must be implemented and validated.
- Results must be logged to WandB.
🛠️ Project Management
- Assign the project to the Fast-LLM project.
- Set the
Estimate
field (in days) in the GitHub project. - Use the
Size
field to categorize the PR size (Small/Medium/Large). - Assign an owner when opening the issue.