[fix] make sure neptune saves the checkpoint

[kshitij12345]

Neptune logger doesn't actually save the checkpoint.

Simple reproducer:

import torch
import tempfile
import neptune
import time


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(8192, 8192)

    def forward(self, x):
        return self.linear(x)


model = Net()

run = neptune.init_run()

start = time.time()
with tempfile.NamedTemporaryFile() as tmp:
    torch.save(model.state_dict(), tmp.file)
    run['model'].upload(tmp.name)  # Fails because (upload is async and once outside of tmp context the file is deleted).

print(time.time() - start)


start = time.time()
with tempfile.NamedTemporaryFile() as tmp:
    torch.save(model.state_dict(), tmp.file)
    # Uploads in chunks.
    run['model_stream'].upload(neptune.types.File.from_stream(tmp.file))  # Works!

print(time.time() - start)

Description:

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[vfdev-5]

Thanks @kshitij12345 !

Can we add a unit test for that somehow ?

[kshitij12345]

Can we add a unit test for that somehow ?

I think this can be verified only by actually running a valid run and verifying that checkpoint was uploaded correctly. Right now, there is mock test which verifies that upload was called but that doesn't guarantee that upload was successful.

[vfdev-5]

Can we add a unit test for that somehow ?

I think this can be verified only by actually running a valid run and verifying that checkpoint was uploaded correctly. Right now, there is mock test which verifies that upload was called but that doesn't guarantee that upload was successful.

I see, I think this is enough in the sense that if we use correctly Neptune API as it is from Neptune side to check whether API is working correctly and uploading artifacts to the server.

[vfdev-5]

@kshitij12345 Kshiteej, can you please run the following to fix code formatting issue:

bash tests/run_code_style.sh install
bash tests/run_code_style.sh fmt

[kshitij12345]

Ah, sorry. I missed that. Updated now. Thanks!