Add pytorch Lightning example

[st7ma784]

Summary

Add Pytorch Lightning Implementation for Classification model. The ambition of this change is to allow rapid developement and evaluation of classifiers using HPC provision, and refactor the framework for scalable deployment.

Describe your changes

The main change is removing the generic pytorch wrappers for iterating over dataloaders, and revising how "phases" is referred to in the code, instead leaning on PTL's implementation of separate train, validation, inference steps. Some other minor refactoring has occurred to remove repeated 'if' statements during each step to aid with pipelining.

Checklist before assigning a reviewer (update as needed)

• Self-review code
• Test with map data
• Update reference docs
• Update changelog
• Add test code to HEC
• Add logging information
• Compare to previous implementation

Reviewer checklist

Please check this conforms to downstream needs in example notebooks.

• Everything looks ok?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 288 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mapreader/classify/pytorchLightningClassifier.py	0.00%	288 Missing ⚠️

Flag	Coverage Δ
unittests	76.42% <0.00%> (-2.55%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
mapreader/classify/pytorchLightningClassifier.py	0.00% <0.00%> (ø)

[rwood-97]

Hi, thanks so much for this.

I'm going to spend some time the rest of this week seeing if I can wrap this code into a class similar to the ClassifierContainer so we can essentially swap over to pytorch lightning whilst still following same/similar API.

Do you know if there is any reason we wouldn't want to swap over completely? (i.e. what are downsides of using lightning vs just normal pytorch?)

[st7ma784]

The short and sweet of it is that there's no reason other than compatability with the way the rest of your repository works. I had a quick glance through some of the example notebooks, which might need a bit of tailoring to just adjust how the training is called. Other than that should be all good. I would suggest just using the simple script as is to test different models, or finding the best hyperparameters to then put back into the wider workflow. Glad it's useful. Let me know if you have any questions about ways to test this using HPC. Best, Steve

…

________________________________ From: Rosie Wood ***@***.***> Sent: Tuesday, January 7, 2025 1:18 PM To: maps-as-data/MapReader ***@***.***> Cc: Mander, Stephen (manders3) ***@***.***>; Author ***@***.***> Subject: [External] Re: [maps-as-data/MapReader] Add pytorch Lightning example (PR #545) This email originated outside the University. Check before clicking links or attachments. Hi, thanks so much for this. I'm going to spend some time the rest of this week seeing if I can wrap this code into a class similar to the ClassifierContainer so we can essentially swap over to pytorch lightning whilst still following same/similar API. Do you know if there is any reason we wouldn't want to swap over completely? (i.e. what are downsides of using lightning vs just normal pytorch?) — Reply to this email directly, view it on GitHub<#545 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI25TU4QSRR7DK7V3ZAURZ32JPHYXAVCNFSM6AAAAABUVZ56OCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZVGI3TOMJRGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>