Skip to content

Add s5cmd progress bar #4204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 22, 2025
Merged

Add s5cmd progress bar #4204

merged 1 commit into from
Aug 22, 2025

Conversation

RodrigoVillar
Copy link
Contributor

Why this should be merged

When downloading large objects from s3, the following is often shown:

Screenshot 2025-08-22 at 15 15 34

The repeating text at the bottom isn't useful, however, as it doesn't show the current progress towards downloading the s3 object. Using the --show-progress flag, however, reveals the current download progress, which I find extremely useful for large object downloads:

Screenshot 2025-08-22 at 15 16 38

How this works

Adds the --show-progress flag to the s5cmd cp command.

How this was tested

CI + ran import-s3-to-dir task locally.

Need to be documented in RELEASES.md?

No

@RodrigoVillar RodrigoVillar marked this pull request as ready for review August 22, 2025 19:45
@Copilot Copilot AI review requested due to automatic review settings August 22, 2025 19:45
Copilot
Copilot AI reviewed Aug 22, 2025
View reviewed changes
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a progress bar to S3 operations by enabling the --show-progress flag in the s5cmd copy command. This provides visual feedback on download progress for large S3 objects, replacing repetitive text output with a useful progress indicator.

  • Adds --show-progress flag to s5cmd cp command for better user experience during large file transfers

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@joshua-kim joshua-kim enabled auto-merge August 22, 2025 20:03
@joshua-kim joshua-kim changed the title feat: add s5cmd progress bar Add s5cmd progress bar Aug 22, 2025
@joshua-kim joshua-kim added this pull request to the merge queue Aug 22, 2025
Merged via the queue into master with commit b18ffc1 Aug 22, 2025
31 checks passed
@joshua-kim joshua-kim deleted the RodrigoVillar/s5cmd-progress branch August 22, 2025 20:51
@github-project-automation github-project-automation bot moved this to Done 🎉 in avalanchego Aug 22, 2025
joshua-kim added a commit that referenced this pull request Sep 3, 2025
@joshua-kim
Squashed commit of the following:
74d60c2 
commit 66ca7dc
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Tue Sep 2 16:49:55 2025 -0400

    feat(load): add firewood flag (#4235)

commit 274541b
Author: Suyan Qu <36519575+qusuyan@users.noreply.github.com>
Date:   Tue Sep 2 14:10:48 2025 -0500

    feat: add parameters to disable metrics (#4214)

commit e03af84
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Tue Sep 2 12:30:07 2025 -0400

    Add timeout parameter to C-Chain re-execution jobs (#4223)

commit 0e20485
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Tue Sep 2 11:58:29 2025 -0400

    Comment out schedule trigger for re-execution on w/container (#4234)

    Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 847eba1
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Fri Aug 29 14:38:11 2025 -0400

    Add back empty schedule entry for reexecute w/ container job (#4230)

commit a958b8a
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Fri Aug 29 12:56:06 2025 -0400

    Add newline to workflow dispatch (#4229)

    Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>

commit 7ec2258
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Thu Aug 28 17:11:38 2025 -0400

    Push benchmark re-execute results on master workflow dispatch (#4224)

commit 34f983e
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Thu Aug 28 15:33:12 2025 -0400

    Disambiguate source vs exec variable names in reexecute tasks (#4200)

    Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 99578a2
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Thu Aug 28 12:52:31 2025 -0400

    Write grafana link to logs and github step summary (#4219)

commit 814300c
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Thu Aug 28 12:37:05 2025 -0400

    Remove firewood entry from PR triggers due to flakes (#4227)

commit 40fbcd5
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Thu Aug 28 00:24:54 2025 -0400

    refactor(load): simulator contract (#4181)

commit 6195e1f
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Wed Aug 27 17:31:51 2025 -0400

    chore: remove unzip mention (#4226)

commit 59e88f3
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Wed Aug 27 11:17:27 2025 -0400

    Remove schedule trigger for w/ container job that evaluates to empty matrix (#4218)

commit c2563d1
Author: Stephen Buttolph <stephen@avalabs.org>
Date:   Tue Aug 26 19:07:47 2025 -0400

    Update versions for v1.13.5 (#4217)

commit a0ccd66
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Tue Aug 26 12:34:54 2025 -0400

    Add support for passing config and predefined configs to VM re-execution tests (#4180)

commit cc3242f
Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Date:   Mon Aug 25 18:49:28 2025 -0400

    Dynamically update mempool gossip request rate limit (#4162)

    Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
    Co-authored-by: Stephen Buttolph <stephen@avalabs.org>

commit f2e3273
Author: Draco <draco@dracoli.com>
Date:   Mon Aug 25 15:00:56 2025 -0400

    Add ability to create zstd compressor with compression level (#4203)

commit 441f441
Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Date:   Mon Aug 25 12:11:07 2025 -0400

    Remove buf lint action (#4189)

    Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

commit 4bcb221
Author: Stephen Buttolph <stephen@avalabs.org>
Date:   Sat Aug 23 15:43:06 2025 -0400

    Update block + validator + pgo checkpoints to 2025-08-23 (#4205)

commit b18ffc1
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Fri Aug 22 16:34:53 2025 -0400

    Add s5cmd progress bar (#4204)

commit 2100bee
Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com>
Date:   Fri Aug 22 11:52:31 2025 -0400

    Rename Engine Types (#4193)

    Signed-off-by: Sam Liokumovich <65994425+samliok@users.noreply.github.com>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 33727a8
Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Date:   Fri Aug 22 11:00:12 2025 -0400

    Count throttled requests as hits (#4199)

    Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

commit b939be4
Author: Draco <draco@dracoli.com>
Date:   Thu Aug 21 14:22:54 2025 -0400

    fix: blockdb file eviction race issue (#4186)

commit 778ccfe
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Thu Aug 21 11:40:03 2025 -0400

    Add config option for AWS S3 read only credential duration (#4192)

commit ae41355
Author: Stephen Buttolph <stephen@avalabs.org>
Date:   Wed Aug 20 16:46:49 2025 -0400

    Add redundant import alias linting (#4191)

    Signed-off-by: Stephen Buttolph <stephen@avalabs.org>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit a3b5c6a
Author: Stephen Buttolph <stephen@avalabs.org>
Date:   Wed Aug 20 10:53:24 2025 -0400

    Make Draco the codeowner of the blockdb (#4187)

commit a24ac68
Author: queryfast <113781357+queryfast@users.noreply.github.com>
Date:   Wed Aug 20 22:18:21 2025 +0800

    refactor: replace []byte(fmt.Sprintf) with fmt.Appendf (#4161)

    Signed-off-by: queryfast <queryfast@outlook.com>

commit 7aa6a17
Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com>
Date:   Tue Aug 19 14:39:40 2025 -0400

    Rename height field to numBlocks (#4184)

commit 7d7e1fe
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Tue Aug 19 13:24:59 2025 -0400

    Add optional step to archive post-reexecution state to S3 (#4172)

    Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit ebe0558
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Tue Aug 19 12:11:34 2025 -0400

    Change cache path to tmp included in gitignore (#4183)

commit e5593be
Author: Draco <draco@dracoli.com>
Date:   Tue Aug 19 12:01:43 2025 -0400

    Block Database (#4027)

commit 940b96f
Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com>
Date:   Tue Aug 19 11:36:37 2025 -0400

    Storage Component For Simplex (#4122)

    Signed-off-by: Sam Liokumovich <65994425+samliok@users.noreply.github.com>

commit 6d7e2dc
Author: Nicolas Arnedo Villanueva <nicolasarnedo99@gmail.com>
Date:   Tue Aug 19 16:59:58 2025 +0200

    `config/config.md:` Added Env Variable representation of flags + improved UI design (#4110)

    Signed-off-by: Meaghan FitzGerald <meag.fitz@avalabs.org>
    Signed-off-by: Nicolas Arnedo Villanueva <nicolasarnedo99@gmail.com>
    Co-authored-by: Meaghan FitzGerald <meag.fitz@avalabs.org>
    Co-authored-by: Stephen Buttolph <stephen@avalabs.org>

commit 81f13b2
Author: Draco <draco@dracoli.com>
Date:   Mon Aug 18 13:59:43 2025 -0400

    feat: add eviction callback in LRU cache (#4088)

commit 4f5acfc
Author: Jonathan Oppenheimer <jonathan.oppenheimer@avalabs.org>
Date:   Mon Aug 18 13:16:44 2025 -0400

    Migrate predicate package from evm repos (#4147)

    Signed-off-by: Jonathan Oppenheimer <147infiniti@gmail.com>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
    Co-authored-by: Stephen Buttolph <stephen@avalabs.org>
    Co-authored-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

commit 335e79f
Author: Kendra Karol Sevilla <smpkdrew@gmail.com>
Date:   Mon Aug 18 18:45:52 2025 +0200

    chore: fix typo (#4179)

    Signed-off-by: Kendra Karol Sevilla <smpkdrew@gmail.com>

commit 7275b02
Author: yinwenyu6 <yinwenyu6@outlook.com>
Date:   Mon Aug 18 22:29:03 2025 +0800

    chore: fix function name (#4178)

    Signed-off-by: yinwenyu6 <yinwenyu6@outlook.com>

commit 3b0c595
Author: yacovm <yacovm@users.noreply.github.com>
Date:   Mon Aug 18 16:28:29 2025 +0200

    Fix typo in comment - PChainHeight context (#4176)

    Signed-off-by: Yacov Manevich <yacov.manevich@avalabs.org>

commit 96f30d1
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Fri Aug 15 02:15:44 2025 -0400

    feat(load): add token test (#4171)

commit e285ce0
Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com>
Date:   Thu Aug 14 13:52:41 2025 -0400

    Use EmptyVoteMetadata in Simplex Proto Messages (#4174)

commit 5c72544
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Wed Aug 13 10:34:58 2025 -0400

    Move C-Chain benchmark to custom action and add ARC + GH runner triggers (#4165)

commit 3b0f8d4
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Tue Aug 5 20:14:38 2025 -0400

    refactor(load): remove context from test interface (#4157)

commit a893a61
Author: Juan Leon <juan.leon@avalabs.org>
Date:   Tue Aug 5 14:36:59 2025 -0400

    Add @joshua-kim as CODEOWNER to testing-related packages (#4118)

    Signed-off-by: Juan Leon <juan.leon@avalabs.org>

commit be28a8b
Author: Galoretka <galoretochka@gmail.com>
Date:   Mon Aug 4 22:39:41 2025 +0300

    chore: fix a typo in gossip,go (#4154)

    Signed-off-by: Galoretka <galoretochka@gmail.com>

commit b876d78
Author: aaronbuchwald <aaron.buchwald56@gmail.com>
Date:   Mon Aug 4 11:58:22 2025 -0400

    Separate re-execution job params for PR from schedule (#4151)

commit 752e12f
Author: Stephen Buttolph <stephen@avalabs.org>
Date:   Fri Aug 1 16:23:01 2025 -0400

    Update coreth to v0.15.3-rc.5 (#4153)

commit 3ba5246
Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Date:   Fri Aug 1 14:59:24 2025 -0400

    fix metrics tests (#4146)

    Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

commit 0cb887b
Author: Afounso Souza <drewsmpk@gmail.com>
Date:   Fri Aug 1 16:37:53 2025 +0200

    Typo fix (#4150)

    Signed-off-by: Afounso Souza <drewsmpk@gmail.com>

commit 110807a
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Thu Jul 31 22:06:40 2025 -0400

    docs: load (#4132)

commit 24a051a
Author: Jonathan Oppenheimer <jonathan.oppenheimer@avalabs.org>
Date:   Thu Jul 31 19:06:15 2025 -0400

    uplift: Add combined metrics package from evm repositories (#4135)

    Signed-off-by: Jonathan Oppenheimer <147infiniti@gmail.com>
    Co-authored-by: Stephen Buttolph <stephen@avalabs.org>

commit d9b512e
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Thu Jul 31 11:52:39 2025 -0400

    Parameterize values in transfer tests (#4144)

commit 6947e4c
Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com>
Date:   Wed Jul 30 12:27:45 2025 -0400

    feat(load): add trie stress test (#4137)

Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@joshua-kim joshua-kim joshua-kim approved these changes

@aaronbuchwald aaronbuchwald aaronbuchwald approved these changes

@maru-ava maru-ava Awaiting requested review from maru-ava maru-ava is a code owner

Assignees
No one assigned
Labels
None yet
Projects
avalanchego
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@RodrigoVillar @joshua-kim @aaronbuchwald