Provide option to force_delete for GCSToBigQueryOperator

[nathadfield]

When loading data into a BigQuery table, although there are options to create or truncate the destination table using CREATE/WRITE_DISPOSITION, it might also be desirable to recreate the table as part of the task but this is not currently possible and requires a separate task using BigQueryDeleteTableOperator.

Adding a force_delete parameter that simply calls the BigQuery hook's delete_table function would enable this.

[shahar1]

I'm not strongly against, but why not using the existing the operator for that? (I'm questioning the atomicity of transfer operators in general)

[nathadfield]

I'm not strongly against, but why not using the existing the operator for that? (I'm questioning the atomicity of transfer operators in general)

It's a fair question.

The thought process came about because of a scenario I have encountered revolving around the use of BigQuery dataset expiration policies that will automatically drop tables after a specified amount of time, e.g. 7 days, which we do for temporary/staging areas.

Now, suppose I use the GCSToBQOperator with CREATE_IF_NEEDED to load some data followed by another task to perform a query against it, initially this will result in a table being created that will expire exactly 7 days after it was created.

On that seventh day, if everything all runs at the same time, then the table will not have expired yet so the GCSToBQ task will succeed but not recreate the table. However, in the few seconds between this task ending and the downstream task starting it will be deleted resulting in a task failure due to the table not existing.

The current solution to this is to add a prior task using BigQueryDeleteTableOperator which is perfectly viable but just results in lots of extra tasks. Ideally there would be a another CREATE_DISPOSITION option in BigQuery - ALWAYS_RECREATE? - which would achieve the same outcome.

[shahar1]

I'm not strongly against, but why not using the existing the operator for that? (I'm questioning the atomicity of transfer operators in general)

It's a fair question.

The thought process came about because of a scenario I have encountered revolving around the use of BigQuery dataset expiration policies that will automatically drop tables after a specified amount of time, e.g. 7 days, which we do for temporary/staging areas.

Now, suppose I use the GCSToBQOperator with CREATE_IF_NEEDED to load some data followed by another task to perform a query against it, initially this will result in a table being created that will expire exactly 7 days after it was created.

On that seventh day, if everything all runs at the same time, then the table will not have expired yet so the GCSToBQ task will succeed but not recreate the table. However, in the few seconds between this task ending and the downstream task starting it will be deleted resulting in a task failure due to the table not existing.

The current solution to this is to add a prior task using BigQueryDeleteTableOperator which is perfectly viable but just results in lots of extra tasks. Ideally there would be a another CREATE_DISPOSITION option in BigQuery - ALWAYS_RECREATE? - which would achieve the same outcome.

Sounds fine by me, I'd be happy for additional feedback before merging.

[potiuk]

LGTM