Skip to content

Delete ESS checkfiles on the server too, not just locally - #1000

Open
alongd wants to merge 1 commit into
mainfrom
remote-check-cleanup
Open

Delete ESS checkfiles on the server too, not just locally#1000
alongd wants to merge 1 commit into
mainfrom
remote-check-cleanup

Conversation

@alongd

@alongd alongd commented Aug 19, 2026

Copy link
Copy Markdown
Member

ARC uploads a Gaussian checkpoint as a wavefunction guess for each job and downloads it again afterwards. When a run terminates it deletes those files — but only on the local machine:

  • delete_check_files(project_directory) walks <project_directory>/calcs;
  • main.py calls it unconditionally at the end of a run, since keep_checks defaults to False;
  • no code path anywhere deletes anything on the remote server.

So the documented policy — "they usually take up lots of space and are not needed after ARC terminates" — is half-implemented, and every checkpoint ARC has ever uploaded is still sitting in the remote project directory.

Measured impact. On a quota-managed cluster, 82.3% of a fully characterised species' 61.4 MiB footprint is exactly this class of file — the next largest class is 5.6× smaller — and 6.35 GiB had silently accumulated against a 300 GB quota. Nothing downstream reads the remote copy: thermochemistry is parsed from the primary output log, and the local copy is the one reused as a guess.

What this does. Adds SSHClient.delete_check_files and a delete_remote_check_files helper, and routes the existing teardown through a new ARC.clean_check_files. Design constraints kept deliberately tight:

  • keep_checks=True keeps them everywhere, local and remote — the flag's meaning is unchanged;
  • deletion is scoped to *.chk under the run's own remote project directory, using the same path construction job/adapter.py already uses, and no-ops if that directory does not exist;
  • errors are logged and never raised. This runs after the science is done; an unreachable server at teardown is an inconvenience, not a reason to lose a run;
  • servers named local are skipped; nothing about uploads, the wavefunction-guess mechanism, or keep_checks's default changes.

Tests cover all three: deleted with keep_checks=False (asserting only .chk, and only under this project's directory), kept with keep_checks=True, and a run completing normally when the server call raises. Verified as a regression test rather than a tautology — removing just the remote call while leaving the helper in place makes test_check_files_are_deleted_locally_and_remotely fail.

Comment thread arc/main_test.py Fixed
Comment thread arc/job/ssh.py Outdated
"""
path = servers[server].get('path', '').lower()
path = os.path.join(path, servers[server]['un']) if path else ''
return os.path.join(path, 'runs', 'ARC_Projects', project)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not alwyas running under 'runs', 'ARC_Projects'

Comment thread arc/job/ssh.py Outdated
command = f'chmod{recursive} {mode} {file_name}'
self._send_command_to_server(command, remote_path)

def delete_check_files(self, remote_path: str) -> None:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the difference between this and arc.common.delete_check_files()?

ARC's `keep_checks` policy was only half implemented: `delete_check_files()`
walked the local `calcs` tree, and no code path ever removed the check files
ARC had uploaded to a server. They accumulate silently -- measured at 82% of a
fully characterised species' remote footprint, against a managed quota.

`ARC.clean_check_files()` now owns both sides of the policy: with `keep_checks`
false it deletes the local check files as before and then, on every non-local
server in `ess_settings`, deletes `*.chk` under that project's own remote
directory. `keep_checks=True` keeps them everywhere. Cleanup runs once ARC is
done with the science, so a server that cannot be reached at that point is
logged and skipped rather than failing the run.
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.47%. Comparing base (d033cbb) to head (90d4838).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1000      +/-   ##
==========================================
+ Coverage   64.38%   64.47%   +0.08%     
==========================================
  Files         119      119              
  Lines       39601    39627      +26     
  Branches    10269    10275       +6     
==========================================
+ Hits        25499    25549      +50     
+ Misses      11121    11089      -32     
- Partials     2981     2989       +8     
Flag Coverage Δ
functionaltests 64.47% <ø> (+0.08%) ⬆️
unittests 64.47% <ø> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants