Add to_arrow to get a pyarrow.Table from query results.
#8609
Conversation
An Arrow `Table` supports a richer set of types than a pandas `DataFrame`, and is the basis of many data analysis systems. It can be used in conjunction with pandas through the `Table.to_pandas()` method or the pandas extension types provided by the `fletcher` package.
googlebot
added
the
cla: yes
There was a problem hiding this comment.
None of the tests seem to deal with NULL interactions. Might be good to add something? From a brief read of pyarrow docs, it looks like you don't have to deal with the avro type unioning at schema time, but nulls in arrays vs scalar values looks like it has some differences.
| pyarrow.field("field05", pyarrow.float64()), | ||
| pyarrow.field("field06", pyarrow.float64()), | ||
| pyarrow.field("field07", module_under_test.pyarrow_numeric()), | ||
| pyarrow.field("field08", pyarrow.bool_()), |
There was a problem hiding this comment.
Does the underscore signify anything, or just an arrow oddity in type representation?
There was a problem hiding this comment.
Trailing underscore is the Python convention for avoiding name collisions with Python built-in functions such as bool.
There was a problem hiding this comment.
Approving based on in person conversation. There's possible followup around null handling, but arrow exhibits more nullable-by-default behaviors rather than needing explicit care. BigQuery doesn't allow null elements in an array in results/tables, so the difference in arrow's handling isn't relevant here.
An Arrow
Tablesupports a richer set of types than a pandasDataFrame,and is the basis of many data analysis systems. It can be used in
conjunction with pandas through the
Table.to_pandas()method or thepandas extension types provided by the
fletcherpackage.Towards #5204.