Add checksum validation on artifact upload#1063
Merged
Conversation
robherley
commented
Apr 26, 2022
robherley
commented
Apr 26, 2022
eee8fbe to
d5c547c
Compare
konradpabjan
approved these changes
May 19, 2022
This was referenced May 19, 2022
bors bot
referenced
this pull request
in OpenPoolProject/stratum
Nov 6, 2022
270: chore(deps): update actions/upload-artifact action to v3.1.1 r=renovate[bot] a=renovate[bot] [](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/upload-artifact](https://togithub.com/actions/upload-artifact) | action | minor | `v3.0.0` -> `v3.1.1` | --- ### Release Notes <details> <summary>actions/upload-artifact</summary> ### [`v3.1.1`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.1) [Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.1.0...v3.1.1) - Update actions/core package to latest version to remove `set-output` deprecation warning [#​351](https://togithub.com/actions/upload-artifact/issues/351) ### [`v3.1.0`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.0) [Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.0.0...v3.1.0) ##### What's Changed - Bump [`@​actions/artifact](https://togithub.com/actions/artifact)` to v1.1.0 ([https://github.com/actions/upload-artifact/pull/327](https://togithub.com/actions/upload-artifact/pull/327)) - Adds checksum headers on artifact upload ([https://github.com/actions/toolkit/pull/1095](https://togithub.com/actions/toolkit/pull/1095)) ([https://github.com/actions/toolkit/pull/1063](https://togithub.com/actions/toolkit/pull/1063)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9--> 271: chore(deps): update github/codeql-action action to v1.1.31 r=renovate[bot] a=renovate[bot] [](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github/codeql-action](https://togithub.com/github/codeql-action) | action | minor | `v1.0.26` -> `v1.1.31` | --- ### Release Notes <details> <summary>github/codeql-action</summary> ### [`v1.1.31`](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31) ### [`v1.1.30`](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30) ### [`v1.1.29`](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29) ### [`v1.1.28`](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28) ### [`v1.1.27`](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27) ### [`v1.1.26`](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26) ### [`v1.1.25`](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25) ### [`v1.1.24`](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24) ### [`v1.1.23`](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23) ### [`v1.1.22`](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22) ### [`v1.1.21`](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21) ### [`v1.1.20`](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20) ### [`v1.1.19`](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19) ### [`v1.1.18`](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18) ### [`v1.1.17`](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17) ### [`v1.1.16`](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16) ### [`v1.1.15`](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15) ### [`v1.1.14`](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14) ### [`v1.1.13`](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13) ### [`v1.1.12`](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12) ### [`v1.1.11`](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11) ### [`v1.1.10`](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10) ### [`v1.1.9`](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9) ### [`v1.1.8`](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8) ### [`v1.1.7`](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7) ### [`v1.1.6`](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6) ### [`v1.1.5`](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5) ### [`v1.1.4`](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4) ### [`v1.1.3`](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3) ### [`v1.1.2`](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2) ### [`v1.1.1`](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1) ### [`v1.1.0`](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0) ### [`v1.0.32`](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32) ### [`v1.0.31`](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31) ### [`v1.0.30`](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30) ### [`v1.0.29`](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29) ### [`v1.0.28`](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28) ### [`v1.0.27`](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
|
I ran into this problem with a service that handled large file uploads and the root cause was that our continuous deployment was killing our containers after a shorter than expected timeout. In our case, during deployment, hosts are taken out of the pool and in-flight requests are allowed to finish but it was not known to us at the time that a timeout was even in effect. Also if you're wondering how I even stumbled upon this: https://youtu.be/9qljpi5jiMQ?t=722 |
rafegoldberg
pushed a commit
to readmeio/markdown
that referenced
this pull request
Oct 30, 2024
## Version 7.8.0 ### ✨ New & Improved * compat parser ([#1004](#1004)) ([ead267e](ead267e)) ### 🛠 Fixes & Updates * bold in tables ([#1006](#1006)) ([bc481d9](bc481d9)) * **deps:** bump actions/setup-node from 3 to 4 ([#821](#821)) ([f96ea89](f96ea89)), closes [actions/setup-node#866](actions/setup-node#866) [actions/setup-node#868](actions/setup-node#868) [actions/setup-node#876](actions/setup-node#876) [actions/setup-node#868](actions/setup-node#868) [actions/setup-node#861](actions/setup-node#861) [actions/setup-node#859](actions/setup-node#859) [actions/setup-node#870](actions/setup-node#870) [actions/setup-node#872](actions/setup-node#872) [actions/setup-node#875](actions/setup-node#875) [actions/setup-node#831](actions/setup-node#831) [actions/setup-node#803](actions/setup-node#803) [actions/setup-node#809](actions/setup-node#809) [actions/setup-node#816](actions/setup-node#816) [actions/setup-node#794](actions/setup-node#794) [actions/setup-node#710](actions/setup-node#710) [actions/setup-node#812](actions/setup-node#812) [actions/setup-node#808](actions/setup-node#808) [actions/setup-node#804](actions/setup-node#804) [actions/setup-node#802](actions/setup-node#802) [actions/setup-node#807](actions/setup-node#807) [#876](#876) [#868](#868) [#866](#866) * **deps:** bump actions/upload-artifact from 3 to 4 ([#846](#846)) ([5a72802](5a72802)), closes [actions/upload-artifact#464](actions/upload-artifact#464) [actions/upload-artifact#313](actions/upload-artifact#313) [actions/upload-artifact#436](actions/upload-artifact#436) [#374](#374) [#375](#375) [#351](#351) [actions/upload-artifact#327](actions/upload-artifact#327) [actions/toolkit#1095](actions/toolkit#1095) [actions/toolkit#1063](actions/toolkit#1063) [#515](#515) [#511](#511) [#509](#509) [#505](#505) * **deps:** bump braces from 3.0.2 to 3.0.3 ([#907](#907)) ([6b017a7](6b017a7)), closes [#40](#40) [#37](#37) [#27](#27) * **deps:** bump core-js from 3.36.1 to 3.38.1 ([#1013](#1013)) ([58ceef7](58ceef7)), closes [#1357](https://github.com/readmeio/markdown/issues/1357) [#1361](https://github.com/readmeio/markdown/issues/1361) [#1352](https://github.com/readmeio/markdown/issues/1352) [#1345](https://github.com/readmeio/markdown/issues/1345) [#1344](https://github.com/readmeio/markdown/issues/1344) [#1361](https://github.com/readmeio/markdown/issues/1361) * **deps-dev:** bump css-loader from 6.11.0 to 7.1.2 ([#1015](#1015)) ([6be77cf](6be77cf)), closes [#1600](https://github.com/readmeio/markdown/issues/1600) [#1590](https://github.com/readmeio/markdown/issues/1590) [#1577](https://github.com/readmeio/markdown/issues/1577) [#1600](https://github.com/readmeio/markdown/issues/1600) [#1590](https://github.com/readmeio/markdown/issues/1590) [#1577](https://github.com/readmeio/markdown/issues/1577) [#1600](https://github.com/readmeio/markdown/issues/1600) [#1596](https://github.com/readmeio/markdown/issues/1596) [#1595](https://github.com/readmeio/markdown/issues/1595) [#1591](https://github.com/readmeio/markdown/issues/1591) [#1587](https://github.com/readmeio/markdown/issues/1587) [#1586](https://github.com/readmeio/markdown/issues/1586) * **deps:** bump dset from 3.1.3 to 3.1.4 ([#1018](#1018)) ([be480d7](be480d7)) * **deps:** bump github/codeql-action from 2 to 3 ([#847](#847)) ([4b97fe6](4b97fe6)), closes [#2203](https://github.com/readmeio/markdown/issues/2203) [#2195](https://github.com/readmeio/markdown/issues/2195) [#2185](https://github.com/readmeio/markdown/issues/2185) [#2156](https://github.com/readmeio/markdown/issues/2156) [#2151](https://github.com/readmeio/markdown/issues/2151) [#2147](https://github.com/readmeio/markdown/issues/2147) [#2141](https://github.com/readmeio/markdown/issues/2141) [#2124](https://github.com/readmeio/markdown/issues/2124) [#2121](https://github.com/readmeio/markdown/issues/2121) [/github.com/github/codeql-action/blob/main/#3230---08-jan-2024](https://github.com/readmeio//github.com/github/codeql-action/blob/main//issues/3230---08-jan-2024) [#2106](https://github.com/readmeio/markdown/issues/2106) [#2083](https://github.com/readmeio/markdown/issues/2083) [#2096](https://github.com/readmeio/markdown/issues/2096) [#2073](https://github.com/readmeio/markdown/issues/2073) [#2079](https://github.com/readmeio/markdown/issues/2079) [#2200](https://github.com/readmeio/markdown/issues/2200) [#2198](https://github.com/readmeio/markdown/issues/2198) [#2197](https://github.com/readmeio/markdown/issues/2197) * **deps-dev:** bump jest-puppeteer from 8.0.6 to 10.1.4 ([#1016](#1016)) ([33aa434](33aa434)), closes [#604](#604) [#602](#602) [#599](#599) [#586](#586) [#578](#578) [#576](#576) [#605](#605) * **deps:** bump micromatch from 4.0.5 to 4.0.8 ([#1019](#1019)) ([7be267e](7be267e)), closes [#266](#266) * **deps-dev:** bump mini-css-extract-plugin from 2.8.1 to 2.9.1 ([#1017](#1017)) ([9fb25da](9fb25da)) * **deps-dev:** bump semantic-release from 22.0.12 to 24.2.0 ([#1011](#1011)) ([443c843](443c843)), closes [#3462](https://github.com/readmeio/markdown/issues/3462) [#3460](https://github.com/readmeio/markdown/issues/3460) [#3436](https://github.com/readmeio/markdown/issues/3436) [#3423](https://github.com/readmeio/markdown/issues/3423) [#3462](https://github.com/readmeio/markdown/issues/3462) [#3489](https://github.com/readmeio/markdown/issues/3489) [#3488](https://github.com/readmeio/markdown/issues/3488) [#3486](https://github.com/readmeio/markdown/issues/3486) [#3487](https://github.com/readmeio/markdown/issues/3487) [#3485](https://github.com/readmeio/markdown/issues/3485) [#3483](https://github.com/readmeio/markdown/issues/3483) [#3450](https://github.com/readmeio/markdown/issues/3450) [#3481](https://github.com/readmeio/markdown/issues/3481) [#3424](https://github.com/readmeio/markdown/issues/3424) * **deps-dev:** bump webpack from 5.91.0 to 5.95.0 ([#1014](#1014)) ([b9c8dd2](b9c8dd2)) * update conventional commits ([50a4be9](50a4be9)) * upgrade conventionalcommits ([3646513](3646513)) <!--SKIP CI-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant Issues:
tl;dr adds base64 encodings of the CRC64 and MD5 to the headers of a chunk upload:
Recently we've been running into issues when artifacts are getting corrupted during the upload process. This is extremely rare (seen < 1% during my tests) but happens nonetheless. There still needs to be more investigation as to why these are being corrupted, but we've narrowed the problem areas to be between the upload and file container service handler after isolated testing with md5 checksum headers.
This PR adds a CRC64 and MD5 checksum to the header. Since NodeJS's crypto library (openssl bindings) doesn't have CRC64, I added a simple implementation based on Go's
hash/crc64pkg. Also this is tailored to the CRC polynomial used by azure storage. We're already using this same polynomial places deeper in the stack.I ran some benchmarks and this CRC64 implementation is a bit faster than sha256/md5: