feat: Add FedMD (Federated Model Distillation) support #5826
Conversation
- Add FedMD protocol messages (Tensor, DistillIns, DistillRes, ConsensusIns) - Implement FedMDNumPyClient for client-side logit generation and distillation - Add FedMDStrategy for server-side logit aggregation and consensus - Create public data registry and provider for shared dataset management - Add tensor serialization utilities for protobuf communication - Include comprehensive PyTorch CIFAR-10 example with validation - Add detailed monitoring and convergence analysis tools This implementation enables federated learning through knowledge distillation where clients share logits on public data and learn from aggregated consensus.
- Add explanation-fedmd.rst with detailed FedMD concept explanation - Add tutorial-fedmd-pytorch.rst with complete PyTorch example - Add ref-api-fedmd.rst with full API reference documentation - Update index.rst and reference.rst to include FedMD sections - Include usage examples, configuration options, and best practices - Add troubleshooting guide and customization examples
github-actions
bot
added
the
Contributor
- Add heterogeneous_models.py with different CNN architectures (Small, Medium, Large, ResNet-like) - Add heterogeneous_client.py for creating clients with different models - Add run_heterogeneous_simulation.py for testing heterogeneous FedMD - Support models with 545K to 4.3M parameters - Demonstrate FedMD effectiveness with different model complexities - Show complex models achieve better consensus convergence - Validate heterogeneous federated learning capabilities
|
Hello @kth0910 , thanks for opening the PR to introduce a new example using Also note, that examples need to follow the structure of all other examples (this primarily mean that each example is a Let me know if you need some support. In my view the steps to add both the strategy and examples into flower would be:
|
2 participants
feat: Add FedMD (Federated Model Distillation) support
Issue
Description
Currently, Flower supports traditional federated learning approaches like FedAvg, FedProx, etc., but lacks support for Federated Model Distillation (FedMD). FedMD is an important paradigm that enables knowledge sharing through logit distillation on public data, allowing heterogeneous models to learn from each other without sharing raw data or model parameters.
This approach is particularly valuable for:
Related issues/PRs
This is a new feature request. No related issues found, but this addresses the broader need for more diverse federated learning strategies in Flower.
Proposal
Explanation
This PR introduces comprehensive FedMD support to Flower with the following key components:
Core Protocol & Communication:
Tensor,DistillIns,DistillRes,ConsensusIns) for efficient logit exchangeClient-Side Implementation:
FedMDNumPyClient: Extends NumPyClient with logit generation and distillation capabilitiesServer-Side Strategy:
FedMDStrategy: Handles logit collection, aggregation, and consensus buildingComprehensive Example & Validation:
Key Benefits:
Checklist
#contributions)Any other comments?
This implementation follows Flower's established patterns and integrates seamlessly with existing infrastructure. The example demonstrates successful convergence with clear metrics showing client-consensus distance reduction over training rounds.
The code includes comprehensive error handling, detailed logging, and validation tools to ensure FedMD effectiveness. All new components are properly documented and follow Flower's coding standards.
Example Output: