Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow specifying the content length for resumable uploads #2511

Closed
benjaminp opened this issue Apr 24, 2024 · 2 comments · Fixed by #2661 or #2686
Closed

allow specifying the content length for resumable uploads #2511

benjaminp opened this issue Apr 24, 2024 · 2 comments · Fixed by #2661 or #2686
Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@benjaminp
Copy link

Is your feature request related to a problem? Please describe.
The resumable upload API allows specifying a final length of the object if known in a HTTP header. There is no way to pass this value into the Java GCS API for resumable uploads, though.

Describe the solution you'd like
A Storage.writer method overload that takes the final content length.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/java-storage API. label Apr 24, 2024
@BenWhitehead
Copy link
Collaborator

In general we recommend folks use checksums for validation of all intended bytes reaching gcs rather than relying on the number of bytes. Using a checksum like crc32c will ensure the bytes are received in the correct sequence and match with what GCS receives.

For an example of how the crc32c precondition is provided to an upload, you can take a look at one of our integration tests that verify correct checksum plumbing and handling

storage.writer(
blobInfo, BlobWriteOption.doesNotExist(), BlobWriteOption.crc32cMatch())) {

To compute a crc32c checksum you can use Guava's crc32c HashFunction https://guava.dev/releases/33.1.0-jre/api/docs/com/google/common/hash/Hashing.html#crc32c()

@BenWhitehead BenWhitehead added the type: question Request for information or clarification. Not an issue. label Apr 24, 2024
@benjaminp
Copy link
Author

I agree that a checksum of the content is preferable when available. However, sometimes you are proxying a file from a client that gives you the size upfront but not a checksum in which case GCS verifying the size is better than nothing.

@tritone tritone added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed type: question Request for information or clarification. Not an issue. labels Aug 5, 2024
BenWhitehead added a commit that referenced this issue Aug 6, 2024
Update resumable upload failure detection to be more specific about classifying a request as SCENARIO_5

Fixes #2511
BenWhitehead added a commit that referenced this issue Aug 23, 2024
Update resumable upload failure detection to be more specific about classifying a request as SCENARIO_5

Fixes #2511
BenWhitehead added a commit that referenced this issue Aug 23, 2024
…ns. (#2661)

Update resumable upload failure detection to be more specific about classifying a request as SCENARIO_5

Fixes #2511
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
3 participants