Skip to content

CI-739 - Verify checksum of file #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nathanthorpe merged 10 commits into from
Jul 29, 2025
Merged

CI-739 - Verify checksum of file #161

nathanthorpe merged 10 commits into from
Jul 29, 2025

Conversation

@nathanthorpe
Copy link
Contributor

@nathanthorpe nathanthorpe commented Jul 24, 2025 •
edited
Loading

  • Add file validate method which compares the checksum of the local file with a Cirro file.
  • Replace deprecated MD5 with CRC64, upload files using that.

Example:

portal = DataPortal()
dataset = portal.get_dataset(project_id, dataset_id)

file = dataset.get_file('nucleus_boundaries.parquet')
local_file = Path('~/Downloads/nucleus_boundaries.parquet').expanduser()

file.validate(local_file)

@nathanthorpe nathanthorpe marked this pull request as draft July 24, 2025 21:30
@nathanthorpe nathanthorpe mentioned this pull request Jul 25, 2025
Merged
nathanthorpe
nathanthorpe commented Jul 25, 2025
View reviewed changes
pyproject.toml
miniwdl = {version="^1.13.0", optional=true}
referencing = "^0.36.2"
nf-core = {version = "^3.3.2", optional=true}
awscrt = "==0.27.4"
Copy link
Contributor Author

@nathanthorpe nathanthorpe Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on if this should be optional?

Copy link
Contributor

@wleepang wleepang Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd see uploading data via the SDK/client as a core functionalty, warranting keeping this dependency as installed by default.

@nathanthorpe nathanthorpe marked this pull request as ready for review July 25, 2025 14:11
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 25, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
21.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@nathanthorpe nathanthorpe requested a review from a team July 25, 2025 14:19
@nathanthorpe nathanthorpe merged commit 995a98a into main Jul 29, 2025
10 checks passed
@nathanthorpe nathanthorpe deleted the checksum-validate branch July 29, 2025 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wleepang wleepang wleepang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nathanthorpe @wleepang