add new code about FedRep and Simple_tuning #564
Conversation
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def wrap_FedRepTrainer( | ||
| base_trainer: Type[GeneralTorchTrainer]) -> Type[GeneralTorchTrainer]: | ||
| """ |
There was a problem hiding this comment.
Why delete this explanation?
| @@ -63,6 +63,13 @@ def extend_fl_algo_cfg(cfg): | |||
| cfg.personalization.K = 5 # the local approximation steps for pFedMe | |||
| cfg.personalization.beta = 1.0 # the average moving parameter for pFedMe | |||
|
|
|||
| # parameters for FedRep： | |||
There was a problem hiding this comment.
Please add explanations on those new configs; Please also the explanation of those new configs to the file FederatedScope/federatedscope/core/configs/README.md
| stdv = 1. / math.sqrt(param.size(-1)) | ||
| param.data.uniform_(-stdv, stdv) | ||
| else: | ||
| param.data.uniform_(-stdv, stdv) |
There was a problem hiding this comment.
This else branch implicitly assumes that "weight" para is the first to appear, otherwise stdv will be used before it is assigned a value.
Therefore, it is suggested to add some assertions here or change it to a more robust way of coding to pre-compute stdv
|
|
||
| def wrap_Simple_tuning_Trainer( | ||
| base_trainer: Type[GeneralTorchTrainer]) -> Type[GeneralTorchTrainer]: | ||
|
|
There was a problem hiding this comment.
Add some explanations for the purpose of this wrapper function, and the meaning of Simple_tuning as it is too general.
| def wrap_FedRepTrainer( | ||
| base_trainer: Type[GeneralTorchTrainer]) -> Type[GeneralTorchTrainer]: | ||
|
|
||
| init_FedRep_ctx(base_trainer) |
There was a problem hiding this comment.
Plz add some comments to remind the users that this method should be used in conjunction with cfg.personalization.local_param and cfg.finetune.local_param. Also the basic idea is should be introduced, e.g., which is to
train
cfg.personalization.local_paramparameter in the firstcfg.personalization.epoch_linearrounds, and then train the other remaining parameters in the lastcfg.personalization.epoch_featurerounds.
format change - (rerun-unitest)
fix the import problem
No description provided.