Skip to content

Commit

Permalink
runner.aws_batch: Download .snakemake/metadata/ too
Browse files Browse the repository at this point in the history
Snakemake stores state information per input/output here and uses it to
determine if it needs to re-run rules or not.  It seems akin to the file
mtimes which we already take care to preserve on upload/download.
Additionally, the metadata recorded is used in Snakemake's report
generation and is generally useful for looking at workflow statistics.

Continue to not download all of .snakemake/ en masse because it can
potentially contain files that interfere with local usage and/or are
large and unnecessary.

Resolves: <#373>
Related-to: <nextstrain/docker-base#220>
  • Loading branch information
tsibley committed Jun 17, 2024
1 parent 8ed779c commit 131c55a
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion nextstrain/cli/runner/aws_batch/s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,12 @@ def download_workdir(remote_workdir: S3Object, workdir: Path, patterns: List[str
])

included = path_matcher([
# But we do want the Snakemake logs to come over.
# But we do want the Snakemake logs to come over
".snakemake/log/",

# …and the input/output metadata Snakemake tracks (akin to mtimes,
# which we also preserve).
".snakemake/metadata/",
])

if patterns:
Expand Down

0 comments on commit 131c55a

Please sign in to comment.