Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate to query_and_wait for lower-latency small queries #15

Open
tswast opened this issue Apr 11, 2024 · 2 comments
Open

migrate to query_and_wait for lower-latency small queries #15

tswast opened this issue Apr 11, 2024 · 2 comments
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery-magics API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Collaborator

tswast commented Apr 11, 2024

Similar to googleapis/python-bigquery-pandas#722, let's use query_and_wait to issue queries.

Note: We may want to update query_and_wait to support progress bars on queries that last longer than the first call to jobs.query before doing this.

Thought: I wonder if migrating to use pandas-gbq to issue queries would make the most sense, given the desire to reduce redundancies?

@tswast tswast added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. priority: p3 Desirable enhancement or fix. May not be included in next release. labels Apr 11, 2024
@tswast tswast self-assigned this Apr 11, 2024
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-magics API. label Apr 11, 2024
@tswast tswast added priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. and removed priority: p3 Desirable enhancement or fix. May not be included in next release. api: bigquery Issues related to the googleapis/python-bigquery-magics API. labels Apr 11, 2024
@tswast
Copy link
Collaborator Author

tswast commented Apr 11, 2024

Using read_gbq might also make #16 easier as well, since both code paths could just call the corresponding read_gbq method.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-magics API. label Apr 12, 2024
@tswast tswast assigned chalmerlowe and unassigned tswast Jun 27, 2024
@tswast
Copy link
Collaborator Author

tswast commented Jun 27, 2024

Progress bar is the hardest part. Maybe we add an option to query_and_wait that outputs a link to the job if the query doesn't finish in the first jobs.query API call?

https://github.com/googleapis/python-bigquery-dataframes/blob/014765c22410a0b4559896d163c440f46f7ce98f/bigframes/formatting_helpers.py#L233

for this job linking logic in BigQuery DataFrames.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-magics API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

2 participants