What happened?
Beam 2.72.0 -> 2.75.0 regression.
--gcsEndpoint pipeline option is ignored by GcsUtilV1.
Full PipelineOptions args in my test:
--project=prj-unit-test
--gcsEndpoint=http://<fake-gcs-server>/storage/v1/
--credentialFactoryClass=org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory
Note that no experiments are passed.
Expected:
Beam reads/writes gs:// paths through the configured emulator endpoint.
Actual:
GcsUtilV1 read path calls real GCS:
GET https://storage.googleapis.com/storage/v1/b/result/o/...
Stack:
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/result/o/a%2Fb%2Fdocs%2Fd%2Farchive.zip?fields=contentEncoding,generation,size
{
"code": 403,
"errors": [
{
"domain": "global",
"location": "Authorization",
"locationType": "header",
"message": "The billing account for the owning project is disabled in state absent",
"reason": "accountDisabled"
}
],
"message": "The billing account for the owning project is disabled in state absent"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:479)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:565)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.fetchInitialMetadata(GoogleCloudStorageReadChannel.java:226)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.<init>(GoogleCloudStorageReadChannel.java:176)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$3.<init>(GoogleCloudStorageImpl.java:696)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:695)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:664)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1.open(GcsUtilV1.java:606)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1.open(GcsUtilV1.java:575)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.open(GcsUtil.java:194)
at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.open(GcsFileSystem.java:160)
<...>
Suspected cause:
After the gcs-connector 3.x compatibility changes, Beam constructs GoogleCloudStorageImpl through GoogleCloudStorageOptions / builder path, but the root URL/service path derived from GcsOptions#getGcsEndpoint() is not propagated into GoogleCloudStorageOptions (here).
Notice what is missing: no setStorageRootUrl(...), no setStorageServicePath(...), and no equivalent propagation from gcsOptions.getGcsEndpoint()
As a result, the Hadoop GCSIO read channel defaults to Storage.DEFAULT_ROOT_URL.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components
What happened?
Beam 2.72.0 -> 2.75.0 regression.
--gcsEndpointpipeline option is ignored by GcsUtilV1.Full
PipelineOptionsargs in my test:Note that no experiments are passed.
Expected:
Beam reads/writes gs:// paths through the configured emulator endpoint.
Actual:
GcsUtilV1 read path calls real GCS:
GET https://storage.googleapis.com/storage/v1/b/result/o/...
Stack:
Suspected cause:
After the gcs-connector 3.x compatibility changes, Beam constructs GoogleCloudStorageImpl through GoogleCloudStorageOptions / builder path, but the root URL/service path derived from GcsOptions#getGcsEndpoint() is not propagated into GoogleCloudStorageOptions (here).
Notice what is missing: no
setStorageRootUrl(...), nosetStorageServicePath(...), and no equivalent propagation fromgcsOptions.getGcsEndpoint()As a result, the Hadoop GCSIO read channel defaults to Storage.DEFAULT_ROOT_URL.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components