Support for ORC in GCSToBigQueryOperator

[will-lauer]

Description

GCSToBigQueryOperator needs to support ORC files in addition to all the other formats that it supports. Both #12556 and #12557 have requested this in the past, and both have ended up getting closed due to implementation issues (deprecated functions in the implementation), but that doesn't remove the need for this feature. Either GCSToBigQueryOperator should be extended to support ALL the file types that the BQ supports, or the operator as a whole should likely be deprecated.

Use case/motivation

As a pipeline developer, I sometimes need to generate ORC files for other tools. I should not be prevents from loading these files into BQ using the GCSToBQ transport operator, but rather be able to load into BQ regardless of file type.

Related issues

#12556 and #12557 have both been closed due to technical issues with the implementation, not due to the feature not being needed.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Marked as good first issue, but feel free to contribute it if you do not want to wait for someone to volunteer for it.

[kgw7401]

oh i want to get assigned this issue!

[raphaelauv]

you can use the bigqueryinsertjoboperator, it will let you do anything that is possible with bigquery

( GCSToBigQueryOperator is just a basic helper operator )

[will-lauer]

While I agree that the BigQueryInsertJobOperator is a generic operator that allows you to run any query and is a workaround to the missing ORC functionality in the GCSToBigQueryOperator Transfer operator, I think its still an inconsistency and that GCSToBigQueryOperator needs to be be updated to support ORC (and any other format that BQ supports), or just eliminate GCSToBigQueryOperator as unnecessary.