Interrupted Batch downloads can be resumed

[jewettaijfc]

This PR addresses issue #2600

This PR:

• Adds the replace_existing argument to Batch.load(), Batch.download(), and BatchData.load() functions from container.py. This allows these functions to skip over previously downloaded files.
• For these batch functions, the default value of replace_existing is False.
• A single warning is generated if it looks like any files have already been downloaded. It explains how to override the default behavior (using replace_existing==True).
• I refrained from making any modifications which did not address issue 2600, except:
• I followed Tyler's advice and moved the call to Batch.download() out of Batch.run() and into Batch.load().
• BatchData.load_sim_data() (which downloads only one file) now invokes web.load() with replace_existing=True, instead of False. (This change was made to be consistent with the other single-file download functions. See below.)

Single-file download functions overwrite existing files

There are also at least 10 different functions in container.py, webapi.py, and s3utils.py which download only a single simulation file. After this PR, all of these single-file functions have the opposite behavior: They will overwrite a previously downloaded file. (Previously, some did and others did not.)

I think this should be the default behavior for functions that download single files: It is the responsibility of the user to decide whether or not to download the file.

Discussion

For batch simulations, I think the situation is different because it's unreasonable to ask users to specify all of the files they want to download manually. That's why I only make it possible to automatically skip previously downloaded batch simulation files.

How to test this code

This code is particularly difficult code to write a unit test for (since it requires mocking in a multi-threaded environment). Instead I tested it manually.

Test strategy

• You cannot test this code in this branch because currently the web-API is not working. Ideally, you should rebase the changes from this PR onto a working release branch (v2.9.0rc1). (Unfortunately I had trouble rebasing this PR onto v2.9.0rc1 due to a large number of conflicts in unrelated files.)
• Fortunately all the changes in this PR are in one file (container.py). So we just need to copy that file over to the v2.9.0rc1 version and run the tests. (See details below.)

Detailed test instructions:

git checkout jewettaijfc/issue2600  # checkout this branch
cp -f tidy3d/web/api/container.py /tmp/
git checkout v2.9.0rc1  # web.api works in this branch
cp -f /tmp/container.py tidy3d/web/api/  # (this effectively rebases on v2.9.0rc1)

Then follow the instructions in this file: test_pr2654.py.zip

[greptile-apps]

Reviewing changes made in this pull request

[yaugenst-flex]

Thanks @jewettaijfc! Left some (mostly minor) comments.

[github-actions]

Diff Coverage

Diff: origin/develop...HEAD, staged and unstaged changes

• tidy3d/web/api/container.py (90.9%): Missing lines 469,940

Summary

• Total: 22 lines
• Missing: 2 lines
• Coverage: 90%

tidy3d/web/api/container.py

  465         """
  466 
  467         batch_file = Batch._batch_path(path_dir=path_dir)
  468         batch = Batch.from_file(batch_file)
! 469         return batch.load(path_dir=path_dir, replace_existing=replace_existing)
  470 
  471 
  472 class Batch(WebContainer):
  473     """

---

  936             for task_name, job in self.jobs.items():
  937                 job_path_str = self._job_data_path(task_id=job.task_id, path_dir=path_dir)
  938                 if os.path.exists(job_path_str):
  939                     if replace_existing:
! 940                         log.info(f"File '{job_path_str}' already exists. Overwriting.")
  941                     else:
  942                         log.info(f"File '{job_path_str}' already exists. Skipping.")
  943                         continue
  944                 if "error" in job.status:

---

[jewettaijfc]

I tested the latest version using test_pr2654.py.zip.

The new log.warning() message contains a link to the list of skipped files.

This PR is ready for review again.

[yaugenst-flex]

Thanks @jewettaijfc! Minor comment about argument defaults but looks almost ready to go.

[yaugenst-flex]

LGTM!