GCP Dataproc suddently not accepting current config default values

[sudohainguyen]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-google          8.6.0

Apache Airflow version

2.3.2

Operating System

ubuntu 20.04

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

The incident occurred 2 days ago, all of DataprocCreateCluster thrown this error

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Network 'default' requires explicit subnetworks for instance creation. Specify a 'legacy' or 'auto' network, or specify subnetwork explicitly."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.217.194.95:443 {grpc_message:"Network \'default\' requires explicit subnetworks for instance creation. Specify a \'legacy\' or \'auto\' network, or specify subnetwork explicitly.", grpc_status:3, created_time:"2022-12-30T09:48:37.000807792+00:00"}"

What you think should happen instead

After a few tries with gcloud and pure python client, I found out it was not because of code implementation, maybe on server side, GCP had a few changes.

How to reproduce

1. Create ClusterGenerator object with a few required params, left the rest as default
2. Create a DAG containing DataprocCreateClusterOperator with the config
3. Execute the DAG

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sudohainguyen]

my team resolved this issue by assigning specific value to subnetwork_uri attributes like this

cluster = {
        "project_id": project_id,
        "cluster_name": cluster_name,
        "config": {
...
            "gce_cluster_config": {
                "subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
            },
...
        },
    }

[sudohainguyen]

my team resolved this issue by assigning specific value to subnetwork_uri attributes like this

cluster = {
        "project_id": project_id,
        "cluster_name": cluster_name,
        "config": {
...
            "gce_cluster_config": {
                "subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
            },
...
        },
    }

so my question is, should we put it as default value of ClusterGenerator on behalf of airflow users?

[potiuk]

I believe so, but I think this is more question to Dataproc team - seem that they have changed the API. cc: @bhirsz @bjankie1 @VladaZakharova - and whether there is somethign to be done in the google provider about it.