Skip to content

Multi-Dataset Validation with LM-Loss #65

@tscholak

Description

@tscholak

🎯 Goal (What & Why)

Improve Fast-LLM’s validation pipeline by enabling multiple independent validation datasets, allowing better insights into model performance across different domains. This will streamline evaluations, reduce manual steps, and align with best practices in modern LLM training.

🚀 Execution Plan

Step: Implement Multi-Dataset Validation (LM-Loss/Perplexity)

  • Modify Fast-LLM’s training config to allow defining multiple validation datasets.
  • During validation, compute LM loss (cross-entropy/perplexity) separately for each dataset.
  • Ensure results are logged independently in WandB.

📌 Acceptance Criteria (Must-Haves for Completion)

  • Multi-dataset LM-loss evaluation must be implemented and validated.
  • Results must be logged to WandB.

🛠️ Project Management

  • Assign the project to the Fast-LLM project.
  • Set the Estimate field (in days) in the GitHub project.
  • Use the Size field to categorize the PR size (Small/Medium/Large).
  • Assign an owner when opening the issue.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions