Skip to content

Commit

Permalink
upload-to-s3: Use GNU coreutils sha256sum
Browse files Browse the repository at this point in the history
The GNU coreutils `sha256sum` is much faster than the custom Python
implementation in this repo. We no longer have to worry about
portability since GN coreutils has been added to our conda runtime.¹

¹ <nextstrain/conda-base#70>
  • Loading branch information
joverlee521 committed Jun 14, 2024
1 parent c94d78d commit b67d698
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 16 deletions.
15 changes: 0 additions & 15 deletions sha256sum

This file was deleted.

7 changes: 6 additions & 1 deletion upload-to-s3
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ main() {
local key="${s3path#*/}"

local src_hash dst_hash no_hash=0000000000000000000000000000000000000000000000000000000000000000
src_hash="$("$bin/sha256sum" < "$src")"
src_hash="$(shasum "$src")"
dst_hash="$(aws s3api head-object --bucket "$bucket" --key "$key" --query Metadata.sha256sum --output text 2>/dev/null || echo "$no_hash")"

if [[ $src_hash != "$dst_hash" ]]; then
Expand Down Expand Up @@ -75,4 +75,9 @@ content-type() {
esac
}

shasum() {
# Grab just the hex digest, ignoring the echoed input filename
sha256sum "$@" | cut -d ' ' -f 1
}

main "$@"

0 comments on commit b67d698

Please sign in to comment.