Bug: Dataset not found in US

[danicat]

I'm getting an error while running the BigQuery kfp component to export some data from BQ to GCS. It says: google.api_core.exceptions.NotFound: 404 Not found: xxx was not found in location US which is correct, because the dataset is in EU, but I couldn't find any way to change the default behaviour of looking for datasets in the US.

On our regular codebase we specify the location on the BigQuery client initialisation, but the code here just uses the default: https://github.com/danicat/pipelines/blob/master/component_sdk/python/kfp_component/google/bigquery/_query.py

It is a one line change in the code, but I don't know how to rebuild the ml-pipeline container. The developer guide doesn't mention how to do it (but it does explain how to build containers for other components of the solution).

Please help! :)

[Ark-kun]

the code here just uses the default:

Hmm. I see that there is the dataset_location parameter.
And that parameter is exposed in the component:
https://github.com/danicat/pipelines/blob/b29fbb5041433649335b2ed376940bbb300e3eef/components/gcp/bigquery/query/component.yaml#L40

It is a one line change in the code

We happily accept Pull Requests.

I don't know how to rebuild the ml-pipeline container.

Counter-intuitively, the Dockerfile and the build_image script are here: https://github.com/danicat/pipelines/tree/master/components/gcp/container

[hongye-sun]

@danicat, have you tried the dataset_location parameter of the component? Please let us know your client code if it still fails. Thanks.

[vicaire]

Assuming that this is resolved. Please re-open if not.

[danicat]

Definitely not solved, just didn’t have time to look into this.

[danicat]

@hongye-sun that parameter isn’t used for client instantiation. It’s documented as a dataset output location.

But the existence of that parameter is why I say it would be a trivial change.

[hongye-sun]

I see. I didn't notice that there is a default location setting in the Client. I will set it by reusing the dataset_location.