Model load_from_checkpoint

[ricardorei]

Describe the bug
When loading a model directly from a checkpoint I get an error "OSError: Checkpoint does not contain hyperparameters. Are your model hyperparameters storedin self.hparams?"

But my model clearly has the hparams.

To Reproduce
Just create a model save a checkpoint and try to load it like explained in the documentation:

pretrained_model = MyLightningModule.load_from_checkpoint(
    checkpoint_path='/path/to/pytorch_checkpoint.ckpt'
)

Possible reason
I found that code in the trainer_io.py class line 301:

 try:
     torch.save(checkpoint, filepath)
except AttributeError:
     if 'hparams' in checkpoint:
     del checkpoint['hparams']
     torch.save(checkpoint, filepath)

Obviously if the code to save the checkpoint deletes de hparams the load checkpoint function will not find that...

Expected behavior
A more concise way to easily load a checkpoint without the need for the load_from_metrics function.

[neggert]

IIRC, that was a hack to workaround an edge case where the hparams weren't pickleable. Seems like the original ticket #433 is still open. @williamFalcon do we still need this hack?

[ricardorei]

Maybe the problem is about saving an object with a lamba function. I see this line in the issue log:

File "/private/home/falc/.local/lib/python3.7/site-packages/torch/serialization.py", line 224, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))

Pickle doesn't allow lambda functions to be saved but if it's this the reason, it's an easy fix I believe.

https://stackoverflow.com/questions/25348532/can-python-pickle-lambda-functions

[wakandan]

I'm also having the same issue but I'm not using any lambda

[Borda]

in general lambda function is not serializable, so all these items should be removed before saving
@wakandan @ricardorei @neggert interested in sending PR?

[antvconst]

I've encountered the same problem. Looks like the culprit is this line: https://github.com/williamFalcon/pytorch-lightning/blob/6666ca5af39aa2d3e5a483da3d7f6bb76514cc9f/pytorch_lightning/trainer/trainer_io.py#L321
After a bit of debugging I've figured out that the return of vars actually contains all of the bound methods of hparams! This seems to produce a pickling error. In my case it goes like this:

AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

This exception in its turn gets handled in https://github.com/williamFalcon/pytorch-lightning/blob/6666ca5af39aa2d3e5a483da3d7f6bb76514cc9f/pytorch_lightning/trainer/trainer_io.py#L264
by removing the hparams altogether.

Do we need this vars call at all? The TTNamespace that is normally used here is perfectly picklable on its own.

[neggert]

Ahh, good catch. So this works with an argparse.Namespace, but will fail with a TTNamespace? Want to send a PR that changes it to just pickle the Namespace directly and removes the hacky exception handling?

[expectopatronum]

I just encountered the same issue - does this mean I can't load the models I trained in the past couple of days or is there some workaround until this is fixed?

If you still have access to hparams, here is a quick fix for load_from_checkpoint (I am not suggesting to change the method, just in case someone needs this functionality before it is fixed).

@classmethod
def load_from_checkpoint(cls, checkpoint_path, hparams, map_location=None):
    """
    Primary way of loading model from a checkpoint
    :param checkpoint_path:
    :param map_location: dic for mapping storage {'cuda:1':'cuda:0'}
    :return:
    """

    if map_location is not None:
        checkpoint = torch.load(checkpoint_path, map_location=map_location)
    else:
        checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)

    # try:
    #     ckpt_hparams = checkpoint['hparams']
    # except KeyError:
    #     raise IOError(
    #         "Checkpoint does not contain hyperparameters. Are your model hyperparameters stored"
    #         "in self.hparams?"
    #     )
    # hparams = Namespace(**ckpt_hparams)

    # load the state_dict on the model automatically
    model = cls(hparams)
    model.load_state_dict(checkpoint['state_dict'])

    # give model a chance to load something
    model.on_load_checkpoint(checkpoint)

    return model

This is how to use it:

from model_utils.model_definitions.my_classifier import MyCoolModule
from argparse import Namespace

checkpoint_path='/home/verena/.../checkpoints/_ckpt_epoch_18.ckpt'

hparams = {
"batch_size":32,
...
}
namespace = Namespace(**hparams)

model = MyCoolModule.load_from_checkpoint(checkpoint_path=checkpoint_path, hparams=namespace)

[esadr]

I faced the same issue, thanks @expectopatronum for the workaround, it helps me a lot.

[neggert]

Here's a solution that doesn't require modifying your model (from #599).

model = MyModel(whatever, args, you, want)
checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
model.load_state_dict(checkpoint['state_dict'])

[Ir1d]

Hi guys.. it seems that in my case the load_from_checkpoint function didn't load the params for me..
I had to use the code posted above by @neggert instead. ( manually load_state_dict)
Hope it helps.