Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support JSON data type #698

Open
tilschuenemann opened this issue Nov 13, 2023 · 2 comments
Open

Support JSON data type #698

tilschuenemann opened this issue Nov 13, 2023 · 2 comments
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tilschuenemann
Copy link

First of all: Thank you for maintaining this project, it really is a nice addition to pandas!

Is your feature request related to a problem? Please describe.
GBQ supports the JSON data type, but unfortunately pandas-gbq doesn't allow writing (and I believe reading) from a table that features a JSON data type column.

Describe the solution you'd like

  1. Add support for both reading and writing dataframes that feature JSON columns.
  2. Add support for manually specifying JSON columns in the table schema.

Describe alternatives you've considered
Alternatives are:

  • writing data as a string and then loading it as a JSON inside a SQL query to access its key-values
    • the downside here is that the json object can't be used for early query processing, introducing the conversion overhead
  • using the native google.cloud.bigquery module
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label Nov 13, 2023
@Linchin Linchin added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Nov 14, 2023
@subramad
Copy link

Is there any update on this?

@tswast
Copy link
Collaborator

tswast commented Jul 18, 2024

@chelsea-lin is working on a JSON dtype for https://github.com/googleapis/python-db-dtypes-pandas which is blocking this for read_gbq support. Otherwise, we'd end up returning strings (see: googleapis/python-bigquery#1876 (comment)).

Regarding loads via to_gbq, I believe BigQuery's parquet load jobs don't support JSON columns in the table either, so we'll need to address this as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

5 participants