[feature] Add --initialize-from option

[ervteng]

Proposed change(s)

Often-asked for feature. This PR adds a new CLI option, --initialize-from=<run-id>, that allows a user to specify an old run ID. This will load the weights of the old run ID, but start training from step 0 to the newly specified --run-id=<run-id>. This can be used, e.g. for fine-tuning an existing model on a new environment/reward function.

Note that combining with --resume will load the old model, save to the new run-id, but resume training from the step count of the old model.

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

See: https://forum.unity.com/threads/ml-agents-train-an-already-trained-model.840892/

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

[harperj]

Mostly looks good, had a few small discussion points / suggestions

[harperj]

LGTM

[teelrc]

Hey @ervteng I believe there may be a bug in this PR where the step isn't being reset to zero when using --initialize-from and instead it just resumes from where the model you initialized left off. I created the bug below with repro steps:

#3956

[ervteng]

Hi @teelrc, this is indeed a bug and will be fixed soon. If you want to fix it yourself in the meantime, adding self._set_step(0) at line 141 in tf_policy.py will fix the issue. Thanks for reporting it!