feat: Add multi-turn SFT support

[xingyaoww]

Multi-turn Conversation Fine-tuning Support

Overview

This PR adds support for fine-tuning models on multi-turn conversations, including proper chat template handling and loss masking for assistant responses. The implementation includes support for the OpenHands SFT dataset and handles conversations up to 32k tokens.

Key Features

• New MultiTurnSFTDataset class for handling multi-turn conversations
• Proper chat template integration using HuggingFace's apply_chat_template
• Smart loss masking that targets only assistant responses
• Support for both single-turn and multi-turn training in the same trainer
• Token length limiting and conversation filtering

Implementation Details

1. Dataset:

   • Uses proper chat templates from the model's tokenizer
   • Handles system, user, and assistant messages
   • Supports conversations with multiple turns
   • Filters conversations exceeding token limits

2. Training:

   • Added use_multiturn flag in config
   • Added messages_key for multi-turn data format
   • Maintains backward compatibility with single-turn training
   • Works with existing features (FSDP, sequence parallel, etc.)

3. Examples and Tests:

   • Added OpenHands SFT dataset preprocessing script
   • Added multi-turn training example
   • Added comprehensive unit tests
   • Moved tests to appropriate locations

Usage Example

# Config for multi-turn training
data:
  use_multiturn: true
  messages_key: messages
  max_length: 32000
  truncation: right

##Testing

• Unit tests for dataset functionality
• Integration with existing training pipeline
• Example scripts tested with OpenHands dataset
• Coverage for both single-turn and multi-turn modes

Documentation

• Added comments explaining multi-turn specific features
• Updated config defaults with multi-turn options
• Added example scripts with documentation
• Added preprocessing script with dataset-specific handling

---

ok.. this is another PR mostly done by OpenHands with me messaging it about 10 times..

It is still WIP as I'm testing it on a training job now, will report back if it works

[CLAassistant]

All committers have signed the CLA.

[xingyaoww]

Actually i think this PR is ready. I've been using this PR for a while and trained couple model without issue. Would love review here!

Also, do we really need to sign CLA for openhands-agent? 🤣

[eric-haibin-lin]

Oh sorry I missed this PR. Will take a look

[xingyaoww]

Sorry! Must be some copy-pasta :(

@OpenHands can you help me first merge from main - then help me address these review comments?

[eric-haibin-lin]

one last comment. otherwise looks good to me!

[eric-haibin-lin]

Is there any benchmark result you want to share on specific datasets?

[eric-haibin-lin]

Could u merge main for a fix of megatron tests. And also fix lint with format.sh?

[xingyaoww]

@eric-haibin-lin Actually, OpenHands LM 32B was trained using this PR :)
and it got decent performance on SWE-Bench Verified

https://www.all-hands.dev/blog/introducing-openhands-lm-32b----a-strong-open-coding-agent-model