storage: adapt Netty Reactor HTTP client as GCS storage client #410
Conversation
Notes: * Uses direct memory buffers. Recommend to run Diskless with "-Dio.netty.maxDirectMemory=0" to have the Netty cleaner running. * Has static 96 max connections pool. * Has static 32 worker thread pool. * "SO_KEEPALIVE" set for sockets and keep alive header for HTTP. * Compression disabled, producer compression recommended and compressing again likely not beneficial. * GCS client handles redirects, Netty Reactor client following disabled. * Can use static BoringSSL library to offload SSL to OpenSSL. * Zero-copy until the response handling where direct memory buffer bytes are copied to heap manager byte array.
jjaakola-aiven
force-pushed
the
jjaakola-aiven-gcs-storage-use-reactor-netty-http-client
branch
from
239c92c to
697fdfd
Compare
There was a problem hiding this comment.
Here's some feedback and a suggested approach from Google GCS SDK team.
GCS has a gRPC based API, which the GCS Java SDK supports[1] out of the box. My recommendation would be for them to use that if they are wanting an async transport core where we've already done the work to bridge async to streaming (and sometimes non-blocking) channels and to use less CPU compared to the default NetHttpTransport.
I'm curious what they hope to gain by using netty as the lower level transport layer. Netty can be a great http client, but the GCS java SDK is not async in the vast majority of operations and will block the invoking thread as necessary, additionally we've already done a notable amount of work the past few years to reduce memory usage and unnecessary allocations to the heap.
That said, if I were to attempt something like this, I would attempt to get it working in the test suite[1] we already have for the GCS Java SDK. Create a branch of the repo, then modify HttpStorageOptions.HttpStorageDefault#getDefaultTransportOptions()[2] to return their Netty based transport. Then running the integration test suite mvn -Dmaven.test.skip.exec=true -Penable-integration-tests clean verify
From a superficial standpoint, one challenge they will likely run into is the need for streaming of large amount of bytes. The GCS Java SDK does not have any client side limits on how large objects or their streams can be, this can be a challenge for Netty due to Netty operating primarily on ByteBufs. I know of users who are uploading multiple gigabyte objects on a single stream, and similarly reading many gigabytes in a streaming mode often with application backpressure.
[1] https://cloud.google.com/storage/docs/enable-grpc-api
[2] https://github.com/googleapis/java-storage/tree/main/google-cloud-storage/src/test/java/com/google/cloud/storage
[3] https://github.com/googleapis/java-storage/blob/main/google-cloud-storage/src/main/java/com/google/cloud/storage/HttpStorageOptions.java#L341-L343
Notes: