[BUG] Solve the unpickling error in weight Loading

[phoeenniixx]

There is an Unpickling Error in weight loading of pytorch-forecasting-v1 because the weights are not loaded correctly.
The issue seems to be something related to torch which is leading to fail in CI/CD pipeline.

The issue surfaced after the latest lightning update where they exposed the weights_only param and let torch handle it itself.

Fixes #1998

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@0418a84). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2000   +/-   ##
=======================================
  Coverage        ?   87.02%           
=======================================
  Files           ?      160           
  Lines           ?     9511           
  Branches        ?        0           
=======================================
  Hits            ?     8277           
  Misses          ?     1234           
  Partials        ?        0           

Flag	Coverage Δ
cpu	87.02% <100.00%> (?)
pytest	87.02% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fkiraly]

hm, not yet, right?

[phoeenniixx]

Yes, this PR is incomplete until we find a way to pass weights_only to tuner.

[fkiraly]

would overriding tuner work? Feels dangerous, but if we are forced to?

[fkiraly]

doing this feels dangerous, because this has side effects on everything!

I would not do this. Is there a way we can override only in a call, or override a method in a class?

[phoeenniixx]

I have wrapped trainer.fit here then.

[phoeenniixx]

Wrapping around trainer.fit didn't work as lr_find calls trainer._checkpoint_connector.restore after fit that overrode the weights_only param from fit, so now I have wrapped load_checkpoint instead (its deeper in the traceback call - close to torch.load, but still in the lightning layer). I think this should solve the issue?

[fkiraly]

Interesting find that this works!

Though I do not agree with the pattern - overriding the torch installation imo is a bad idea, as it can have unpredictable side effects, as it would override torch.load for any and every call.

Is there a way to avoid doing that, and instead injecting the override at call level?