Skip to content

Complete Fetch Phase (EXTERNAL_LINKS disposition + ARROW format) #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 380 commits into
base: fetch-json-inline
Choose a base branch
from
Draft

Complete Fetch Phase (EXTERNAL_LINKS disposition + ARROW format) #596

wants to merge 380 commits into from
+2,221 −1,820

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jun 13, 2025 •
edited
Loading

WIP

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

varun-edachali-dbx and others added 30 commits May 28, 2025 03:45
@varun-edachali-dbx
fix: ensure open attribute of Connection never fails
677e66a 
in case the openSession takes long, the initialisation of the session
will not complete immediately. This could make the session attribute
inaccessible. If the Connection is deleted in this time, the open()
check will throw because the session attribute does not exist. Thus, we
default to the Connection being closed in this case. This was not an
issue before because open was a direct attribute of the Connection
class. Caught in the integration tests.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
introduce databricksClient interface and thrift backend implementation
4ec8703 
WARNING: incomplete - result set still aligned heavily with thrift and
the necessity of some defined functions is to be validated

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
change names of ThriftBackend -> ThriftDatabricksClient in tests
6ecb6bf 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
formatting: black + re-organise backend into new dir
fe2ce17 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@jprakash-db @varun-edachali-dbx
Update CODEOWNERS (#562)
568b1f4 
new codeowners

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove un-necessary example change
ebbd150 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
[empty commit] trigger integration tests
aa8af45 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
introduce normalised sessionId and CommandId for (near) complete back…
e7be76b 
…end abstraction

status is still aligned with Thrift

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: Any is not defined
7461715 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: get_session_id_hex() is not defined
b295acd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
command_handle -> command_id in ExecuteResponse
8afc5d5 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: active op handle -> active command id in Cursor
b8f9146 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fixed (most) tests by accounting for normalised Session interface
7c733ee 
have not considered how to fix the protocol version check yet

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: convert command id to operationHandle in status_request
0917ea1 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
decouple session class from existing Connection
3fd2a46 
ensure maintenance of current APIs of Connection while delegating
responsibility

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add open property to Connection to ensure maintenance of existing API
03d3ae7 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
use connection open property instead of long chain through session
fb0fa46 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
trigger integration workflow
e3770cd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: ensure open attribute of Connection never fails
3b8002f 
in case the openSession takes long, the initialisation of the session
will not complete immediately. This could make the session attribute
inaccessible. If the Connection is deleted in this time, the open()
check will throw because the session attribute does not exist. Thus, we
default to the Connection being closed in this case. This was not an
issue before because open was a direct attribute of the Connection
class. Caught in the integration tests.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: de-complicate earlier connection open logic
f1a350a 
earlier, one of the integration tests was failing because 'session was
not an attribute of Connection'. This is likely tied to a local
configuration issue related to unittest that was causing an error in the
test suite itself. The tests are now passing without checking for the
session attribute.
c676f9b

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Revert "fix: de-complicate earlier connection open logic"
98b0dc7 
This reverts commit d6b1b19.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
[empty commit] attempt to trigger ci e2e workflow
afee423 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@saishreeeee @varun-edachali-dbx
PECOBLR-86 improve logging on python driver (#556)
2d24fdd 
* PECOBLR-86 Improve logging for debug level

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* PECOBLR-86 Improve logging for debug level

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* fixed format

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* used lazy logging

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* changed debug to error logs

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* used lazy logging

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

---------

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Revert "Merge remote-tracking branch 'upstream/sea-migration' into de…
a5561e8 
…couple-session"

This reverts commit dbb2ec5, reversing
changes made to 7192f11.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Reapply "Merge remote-tracking branch 'upstream/sea-migration' into d…
0d890a5 
…ecouple-session"

This reverts commit bdb8381.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: separate session opening logic from instantiation
c2aa762 
ensures correctness of self.session.open call in Connection

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
chore: use get_handle() instead of private session attribute in client
fe642da 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: remove accidentally removed assertions
394333c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
generalise open session, fix session tests to consider positional args
ef07acd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
formatting (black)
1ef46cf 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
clean up SeaCloudFetchQueue
961873a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
clean up Queue more
b765e33 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
ease log warnings
7ce9d28 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
move sea stuff into sea/ dir
cdc7f42 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'exec-resp-norm' into cloudfetch-sea
54c7f6d
@varun-edachali-dbx
add back SeaResltSet (to fix)
dd7c410 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix merge artifacts
e116d9b 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add back og test
bddef1f 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
refactors
b404af7 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'exec-resp-norm' into cloudfetch-sea
8cf118f
@varun-edachali-dbx
fix tests
a8004a0 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix types
0d4f49a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
default to case sensitive comparison in client side table type filtering
66c2baa 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
lots of loffing
936713c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
move to new test structure
d426655 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
revert to old script
1fef8f3 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
cloudfetch fix?
ce08e01 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove non cloud fetch form multi chunk test
75505cf 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
merge
c0bf461 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
Revert "Merge branch 'sea-res-set' into fetch-json-inline"
597f116 
This reverts commit 40f79b5, reversing
changes made to 170f339.
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
Revert "merge"
0ae2ab0 
This reverts commit c0bf461, reversing
changes made to 75505cf.
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db will be requested when the pull request is marked ready for review deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db will be requested when the pull request is marked ready for review samikshya-db is a code owner

@jprakash-db jprakash-db Awaiting requested review from jprakash-db jprakash-db will be requested when the pull request is marked ready for review jprakash-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db will be requested when the pull request is marked ready for review jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db will be requested when the pull request is marked ready for review madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb will be requested when the pull request is marked ready for review gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db will be requested when the pull request is marked ready for review jayantsing-db is a code owner

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala vikrantpuppala will be requested when the pull request is marked ready for review vikrantpuppala is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 will be requested when the pull request is marked ready for review shivam2680 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.