Repository navigation
feat(go): add a streaming Spark Connect client - #101
Open
HyukjinKwon wants to merge 4 commits into
Open
HyukjinKwon wants to merge 4 commits into
HyukjinKwon wants to merge 4 commits into
Conversation
HyukjinKwon
force-pushed
the
streaming-connect-client
branch
from
October 12, 2026 07:26
829f4b8 to
1adf7c4
Compare
Replace the vendored spark-connect-go session with a small gRPC client in connectimpl. Each query is one ExecutePlan with a SQL relation root, and results stream one Arrow batch at a time. Releasing a reader before the end sends Interrupt for its operation, since Spark keeps running a non-reattachable operation after its RPC is cancelled. Close bounds ReleaseSession and always closes the channel, and the receive limit is 128 MiB. SQLSTATE now comes from any error with a SQLState method, so sparkbase no longer imports connectclient. Co-authored-by: Isaac <no-reply@databricks.com>
A reader stopped by its caller's context, or by an Arrow decode error, left the Spark operation running: Release only interrupted readers that had not hit any error. Track whether the server ended the operation separately from whether the reader stopped reading. Also restore the host and port checks spark-connect-go did before dialing: reject an empty host and a port outside 0-65535, and keep treating a missing or zero port as 15002. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
streaming-connect-client
branch
from
October 13, 2026 00:12
8839498 to
52f714f
Compare
HyukjinKwon
marked this pull request as ready for review
October 13, 2026 00:12
HyukjinKwon
had a problem deploying
to
aws-emr-serverless
October 15, 2026 01:16 — with 
GitHub Actions
Error
GitHub Actions
Error
Drop the tests that ran the client against an in-process fake Spark Connect server, and keep the ones that need no server. Add validation tests that run against Spark: a row over gRPC's default 4 MiB limit, a result of several Arrow batches, and the status and SQLSTATE of a query on a missing table. The timeouts no longer need to be variables. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
had a problem deploying
to
aws-emr-serverless
October 15, 2026 01:51 — with
GitHub Actions
Error
GitHub Actions
Error
Ruff's newer releases flag the nested with statements. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
requested a deployment
to
aws-emr-serverless
October 15, 2026 03:34 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
Contributor
|
@HyukjinKwon sorry for the delay, this is in my queue, but the ADBC release and some other tasks are taking priority for me right now |
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This replaces the vendored spark-connect-go session behind the Connect backend with a small client in
go/internal/connectimplthat streams results one Arrow batch at a time. On Connect,ExecuteQuerynow reports -1 rows affected, andclient_typechanges from_SPARK_CONNECT_GOtoadbc-driver-spark.Part of #96.
This pull request and its description were written by Isaac.