chore: Adding implmentation of RequestBody and RewindableContent #3363

ShreyasSinha · 2025-10-23T10:09:57Z

Adding RequestBody for MPU which internally uses RewindableContent.
Implementing checksum related functions in RewindableContent.
Refer: Support XML MPU in GCS Java SDK #3348

… checksum support.

BenWhitehead · 2025-10-23T22:17:23Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

+    String getMd5() throws IOException {
+      try (SeekableByteChannel in = Files.newByteChannel(path, StandardOpenOption.READ)) {
+        in.position(readOffset);
+        MessageDigest md = MessageDigest.getInstance("MD5");
+        byte[] buffer = new byte[8192];
+        ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
+        while (in.read(byteBuffer) != -1) {
+          byteBuffer.flip();
+          md.update(byteBuffer);
+          byteBuffer.clear();
+        }
+        return Base64.getEncoder().encodeToString(md.digest());
+      } catch (NoSuchAlgorithmException e) {
+        throw new IOException(e);
+      }
+    }


Do we have to send MD5 to gcs, or can we send just the crc32c?

There's not much use in computing both.

We can do without MD5 for now.
Will add it later if we get a strong pushback from Salesforce.

BenWhitehead · 2025-10-23T22:22:50Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

+      com.google.common.hash.Hasher crc32cHasher = Hashing.crc32c().newHasher();
+      for (ByteBuffer buffer : buffers) {
+        crc32cHasher.putBytes(buffer.duplicate());
+      }
+      int crc32cInt = crc32cHasher.hash().asInt();
+      ByteBuffer bb = ByteBuffer.allocate(4);
+      bb.putInt(crc32cInt);
+      return Base64.getEncoder().encodeToString(bb.array());


This should use our existing crc32c infrastructure and options.

Suggested change

com.google.common.hash.Hasher crc32cHasher = Hashing.crc32c().newHasher();

for (ByteBuffer buffer : buffers) {

crc32cHasher.putBytes(buffer.duplicate());

}

int crc32cInt = crc32cHasher.hash().asInt();

ByteBuffer bb = ByteBuffer.allocate(4);

bb.putInt(crc32cInt);

return Base64.getEncoder().encodeToString(bb.array());

GuavaHasher hasher;

{

Hasher defaultHasher = Hasher.defaultHasher();

if (defaultHasher instanceof NoOpHasher) {

return null;

} else {

hasher = Hasher.enabled();

}

}

Crc32cLengthKnown cumulative = Crc32cValue.zero();

for (ByteBuffer buffer : buffers) {

cumulative = cumulative.concat(hasher.hash(buffer::duplicate));

}

return Utils.crc32cCodec.encode(cumulative.getValue());

BenWhitehead · 2025-10-23T22:24:50Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

+    String getMd5() throws IOException {
+      return "";
+    }
+
+    @Override
+    String getCrc32c() throws IOException {
+      return "";
+    }


These values aren't correct. Empty bytes do have md5 and crc32c values that aren't empty string.

WIP changes. It was a draft PR.

BenWhitehead · 2025-10-23T22:26:38Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

+  abstract String getMd5() throws IOException;
+
+  abstract String getCrc32c() throws IOException;


I'm not sure this is the best place for this logic to live. Especially in the case of the File based one, the entire file would have to be read from disk before processing could proceed. gRPC has streaming checksum support, and JSON should be getting it soon as well.

Let me raise this PR along with UploadParts changes only, it would make it easierin terms of review.

BenWhitehead · 2025-10-23T22:26:59Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
 import java.util.Arrays;
+import java.util.Base64; // Changed from com.google.api.client.util.Base64


This line is new, not changed from a previous value.

WIP changes. It was a draft PR.

BenWhitehead · 2025-10-23T22:27:09Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

      Preconditions.checkArgument(
          offset <= totalLength,
-          "provided offset must be less than or equal to totalLength (%s <= %s)",
+          "provided offset must be less than orequal to totalLength (%s <= %s)",


revert this please

WIP changes. It was a draft PR.

BenWhitehead · 2025-10-23T22:27:35Z

...-storage/src/test/java/com/google/cloud/storage/ITMultipartUploadHttpRequestManagerTest.java

+      assertThrows(
+          HttpResponseException.class,
+          () ->
+              multipartUploadHttpRequestManager.sendAbortMultipartUploadRequest(
+                  endpoint, request, httpStorageOptions));


Not related to this change, please revert here.

WIP changes. It was a draft PR.

BenWhitehead · 2025-10-28T21:58:59Z

google-cloud-storage/src/main/java/com/google/cloud/storage/MultipartUploadClientImpl.java

+  public UploadPartResponse uploadPart(UploadPartRequest request, RequestBody requestBody) {
+    return retrier.run(
+        retryAlgorithmManager.idempotent(),
+        () -> httpRequestManager.sendUploadPartRequest(uri, request, requestBody),


This needs to do dirty state tracking, and to actually rewind the content before a followup invocation. See

java-storage/google-cloud-storage/src/main/java/com/google/cloud/storage/JsonResumableSession.java

Lines 62 to 76 in 763be91

AtomicBoolean dirty = new AtomicBoolean(false);

return retrier.run(

() -> {

if (dirty.getAndSet(true)) {

ResumableOperationResult<@Nullable StorageObject> query = query();

long persistedSize = query.getPersistedSize();

if (contentRange.endOffsetEquals(persistedSize) || query.getObject() != null) {

return query;

} else {

task.rewindTo(persistedSize);

}

}

return task.call();

},

Decoder.identity());

for how this is done for json resumable uploads.

Additionally, this should pass the RewindableContent to sendUploadPartRequest so that this method manages the lifecycle of it and integrates correctly with the retries.

BenWhitehead · 2025-10-28T22:02:27Z

google-cloud-storage/src/main/java/com/google/cloud/storage/RewindableContent.java

    return false;
  }

+  abstract String getCrc32c();


nit: return Crc32cLengthKnown and let the consumer of this value own the logic of encoding the value for the respected use.

For example, in grpc, crc32c values are ints not base64 encoded 4-byte big-endian representation of an unsigned 32-bit integer.

BenWhitehead · 2025-10-28T22:03:39Z

...-cloud-storage/src/main/java/com/google/cloud/storage/MultipartUploadHttpRequestManager.java

+  private void addHeadersForUploadPart(RequestBody requestBody, HttpHeaders headers) {
+    headers.put("x-goog-hash", "crc32c=" + requestBody.getContent().getCrc32c());
+  }


nit: one-line logic-less helper methods don't help

Is there a plan to add more in here?

chore: Adding implmentation of RequestBody and RewindableContent with…

f656a9e

… checksum support.

product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/java-storage API. labels Oct 23, 2025

chore: added RequestBody

a34fc29

BenWhitehead requested changes Oct 23, 2025

View reviewed changes

ShreyasSinha added 5 commits October 28, 2025 09:41

chore: merged 4 in to 5

6dc7f67

chore: removed md5 implementation.

c9551de

chore: fixed typo.

2778948

chore: fixed lint

28c8216

feat: Adding API for UploadParts.

504ce09

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Oct 28, 2025

BenWhitehead requested changes Oct 28, 2025

View reviewed changes

ShreyasSinha closed this Oct 30, 2025

-      com.google.common.hash.Hasher crc32cHasher = Hashing.crc32c().newHasher();
-      for (ByteBuffer buffer : buffers) {
-        crc32cHasher.putBytes(buffer.duplicate());
-      }
-      int crc32cInt = crc32cHasher.hash().asInt();
-      ByteBuffer bb = ByteBuffer.allocate(4);
-      bb.putInt(crc32cInt);
-      return Base64.getEncoder().encodeToString(bb.array());
+      GuavaHasher hasher;
+      {
+        Hasher defaultHasher = Hasher.defaultHasher();
+        if (defaultHasher instanceof NoOpHasher) {
+          return null;
+        } else {
+          hasher = Hasher.enabled();
+        }
+      }
+      Crc32cLengthKnown cumulative = Crc32cValue.zero();
+      for (ByteBuffer buffer : buffers) {
+        cumulative = cumulative.concat(hasher.hash(buffer::duplicate));
+      }
+      return Utils.crc32cCodec.encode(cumulative.getValue());

		abstract String getMd5() throws IOException;

		abstract String getCrc32c() throws IOException;

	AtomicBoolean dirty = new AtomicBoolean(false);
	return retrier.run(
	() -> {
	if (dirty.getAndSet(true)) {
	ResumableOperationResult<@Nullable StorageObject> query = query();
	long persistedSize = query.getPersistedSize();
	if (contentRange.endOffsetEquals(persistedSize) \|\| query.getObject() != null) {
	return query;
	} else {
	task.rewindTo(persistedSize);
	}
	}
	return task.call();
	},
	Decoder.identity());

chore: Adding implmentation of RequestBody and RewindableContent #3363

chore: Adding implmentation of RequestBody and RewindableContent #3363

Uh oh!

Conversation

ShreyasSinha commented Oct 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants