[PECOBLR-361] convert column table to arrow if arrow present #551
+21
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
There was a problem hiding this comment.
Pull Request Overview
This PR updates the behavior of fetching query results by converting a ColumnTable to a PyArrow Table when PyArrow is available. It adds a new end-to-end test to validate that the catalogs() API returns a PyArrow Table and updates the fetchall_arrow() method in the client to merge columnar results when appropriate.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/e2e/test_driver.py | Added test_catalogs_returns_arrow_table to verify arrow table conversion. |
| src/databricks/sql/client.py | Updated fetchall_arrow() to merge ColumnTables and convert the final result where PyArrow is present. |
shivam2680
requested review from
deeksha-db,
samikshya-db,
yunbodeng-db,
jackyhu-db and
benc-db
as code owners
|
can you add more description in the PR to explain why you make this change? Is there any disruption risk of this change? |
Updated description |
jackyhu-db
reviewed
Apr 29, 2025
jprakash-db
reviewed
May 5, 2025
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
shivam2680
changed the title
jprakash-db
approved these changes
May 9, 2025
saishreeeee
pushed a commit
that referenced
this pull request
May 26, 2025
saishreeeee
pushed a commit
that referenced
this pull request
May 26, 2025
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
varun-edachali-dbx
pushed a commit
that referenced
this pull request
May 27, 2025
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
varun-edachali-dbx
pushed a commit
that referenced
this pull request
May 28, 2025
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
varun-edachali-dbx
pushed a commit
that referenced
this pull request
May 28, 2025
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
saishreeeee
added a commit
that referenced
this pull request
May 30, 2025
* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed TelemetryHelper Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * [PECOBLR-361] convert column table to arrow if arrow present (#551) Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update CODEOWNERS (#562) new codeowners Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Enhance Cursor close handling and context manager exception management to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 improve logging on python driver (#556) * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update github actions run conditions (#569) More conditions to run github actions Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed example Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed to doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed self.telemetry close line Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * grouped classes Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * formatting Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added more descriptive comments, put dataclasses in a sub-folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed default attributes ordering Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed file names Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added enums to models folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed telemetry batch size Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> Co-authored-by: Shivam Raj <171748731+shivam2680@users.noreply.github.com> Co-authored-by: Jothi Prakash <jothi.prakash@databricks.com> Co-authored-by: Madhav Sainanee <madhav.sainanee@databricks.com>
saishreeeee
pushed a commit
that referenced
this pull request
Jun 4, 2025
saishreeeee
added a commit
that referenced
this pull request
Jun 4, 2025
* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed TelemetryHelper Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * [PECOBLR-361] convert column table to arrow if arrow present (#551) Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update CODEOWNERS (#562) new codeowners Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Enhance Cursor close handling and context manager exception management to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 improve logging on python driver (#556) * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update github actions run conditions (#569) More conditions to run github actions Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed example Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed to doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed self.telemetry close line Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * grouped classes Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * formatting Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added more descriptive comments, put dataclasses in a sub-folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed default attributes ordering Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed file names Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added enums to models folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed telemetry batch size Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> Co-authored-by: Shivam Raj <171748731+shivam2680@users.noreply.github.com> Co-authored-by: Jothi Prakash <jothi.prakash@databricks.com> Co-authored-by: Madhav Sainanee <madhav.sainanee@databricks.com>
saishreeeee
pushed a commit
that referenced
this pull request
Jun 4, 2025
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
saishreeeee
added a commit
that referenced
this pull request
Jun 4, 2025
* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed TelemetryHelper Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * [PECOBLR-361] convert column table to arrow if arrow present (#551) Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update CODEOWNERS (#562) new codeowners Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Enhance Cursor close handling and context manager exception management to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 improve logging on python driver (#556) * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Update github actions run conditions (#569) More conditions to run github actions Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * Added classes required for telemetry Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed example Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed to doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed self.telemetry close line Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * grouped classes Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * formatting Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed doc string Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added more descriptive comments, put dataclasses in a sub-folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed default attributes ordering Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed file names Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * added enums to models folder Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * removed telemetry batch size Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> Co-authored-by: Shivam Raj <171748731+shivam2680@users.noreply.github.com> Co-authored-by: Jothi Prakash <jothi.prakash@databricks.com> Co-authored-by: Madhav Sainanee <madhav.sainanee@databricks.com> Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
Added logic in
fetchall_arrow()to convertColumnTableresults to apyarrow.Tableifpyarrowis installed, maintaining consistency with behavior prior to version 3.5.0. This behavior makes result set aligned with the docstring offetchall_pyarrowmethodHow is this tested?
Introduced a new test case,
test_catalogs_returns_arrow_table, intests/e2e/test_driver.pyto validate that thefetchall_arrowmethod returns apyarrow.Tablewhen used withcursor.catalogs(). This test is conditionally executed if Arrow support is available.Related Tickets & Documents
Resolves #550