enable cloud fetch by default#260

Merged

shivam2680 merged 2 commits intomainfrom

shivam2680/cloud-fetch-enable

Apr 4, 2025

Contributor

shivam2680 commented Apr 2, 2025 •

edited

Loading

Description

Enable cloud fetch (EXTERNAL_LINKS) by default.

Testing

Benchmark tests

Benchmarking results

Following table demonstrates the improvement in execution times for large queries.

test_case	percent_reduction
large_results_2050mb	56.58973580321896
large_results_1025mb	51.78689883913764
large_results_0500mb	47.729468599033815
large_results_0100mb	40.57221647707687
large_results_0250mb	38.71932974266906
large_results_0050mb	19.751809720785936
large_results_0025mb	-7.421625079974409
large_results_0010mb	-7.752808988764046
large_results_0005mb	-36.91376701966717

We get ~57% deduction in large queries execution time.
For result size > 4.1GB, we cannot even fetch without cloud fetch with default spark configs.
Ref error: ERR Total size of serialized results of 32 tasks (4.1 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.


          enable cloud fetch by default

a2ff487

shivam2680 requested a review from deeksha-db

April 2, 2025 09:55

shivam2680 requested review from andrefurlan-db, jackyhu-db, jadewang-db, rcypher-databricks and yunbodeng-db as code owners

April 2, 2025 09:55


          update mocks for cf

68916f1

jackyhu-db approved these changes

View reviewed changes

shivam2680 merged commit 644d6da into main

2 of 3 checks passed

shivam2680 deleted the shivam2680/cloud-fetch-enable branch

April 4, 2025 08:47

shivam2680 mentioned this pull request

prepare release v1.7.0 #262

Merged

atzoum pushed a commit to rudderlabs/sqlconnect-go that referenced this pull request


          feat(databricks): bump github.com/databricks/databricks-sql-go from 1…

1aefe8d

….6.1 to 1.8.0 (#351)

Bumps
[github.com/databricks/databricks-sql-go](https://github.com/databricks/databricks-sql-go)
from 1.6.1 to 1.8.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sql-go/releases">github.com/databricks/databricks-sql-go's
releases</a>.</em></p>
<blockquote>
<h2>v1.8.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Ensure thrift field IDs stay within range by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/276">databricks/databricks-sql-go#276</a></li>
<li>Add Arrow IPC stream iterator for direct access to Arrow buffer by
<a href="https://github.com/jadewang-db"><code>@jadewang-db</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/279">databricks/databricks-sql-go#279</a></li>
<li>Prepare for v1.8.0 release by <a
href="https://github.com/jadewang-db"><code>@jadewang-db</code></a> in
<a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/280">databricks/databricks-sql-go#280</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/jadewang-db"><code>@jadewang-db</code></a>
made their first contribution in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/279">databricks/databricks-sql-go#279</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/databricks/databricks-sql-go/compare/v1.7.1...v1.8.0">https://github.com/databricks/databricks-sql-go/compare/v1.7.1...v1.8.0</a></p>
<h2>v1.7.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix critical CVE-2024-45337 by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/264">databricks/databricks-sql-go#264</a></li>
<li>Add nil handling for isStagingOperation to handle older DBR versions
by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/266">databricks/databricks-sql-go#266</a></li>
<li>[PECOBLR-402] Update thrift client library after cleaning up unused
fields and structs by <a
href="https://github.com/gopalldb"><code>@gopalldb</code></a> in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/268">databricks/databricks-sql-go#268</a></li>
<li>Add schema to ArrowBatchIterator by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/267">databricks/databricks-sql-go#267</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/gopalldb"><code>@gopalldb</code></a>
made their first contribution in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/268">databricks/databricks-sql-go#268</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/databricks/databricks-sql-go/compare/v1.7.0...v1.7.1">https://github.com/databricks/databricks-sql-go/compare/v1.7.0...v1.7.1</a></p>
<h2>v1.7.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Enable cloud fetch mode by default (<a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/260">databricks/databricks-sql-go#260</a>
by <a
href="https://github.com/shivam2680"><code>@shivam2680</code></a>)</li>
<li>Handle thrift protocol version for conditional feature support
(cloud fetch, LZ4 compression, Arrow support) (<a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/261">databricks/databricks-sql-go#261</a>
by <a
href="https://github.com/shivam2680"><code>@shivam2680</code></a>)</li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/shivam2680"><code>@shivam2680</code></a> made
their first contribution in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/260">databricks/databricks-sql-go#260</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/databricks/databricks-sql-go/compare/v1.6.2...v1.7.0">https://github.com/databricks/databricks-sql-go/compare/v1.6.2...v1.7.0</a></p>
<h2>v1.6.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Support positional query parameters by <a
href="https://github.com/kravets-levko"><code>@kravets-levko</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/247">databricks/databricks-sql-go#247</a></li>
<li>fix: Add custom auth headers into cloud fetch request by <a
href="https://github.com/jackyhu-db"><code>@jackyhu-db</code></a> in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/249">databricks/databricks-sql-go#249</a></li>
<li>fix: Propagate error in TFetchResults by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/255">databricks/databricks-sql-go#255</a></li>
<li>fix(security): GO-2024-2947, update to
github.com/hashicorp/go-retryablehttp@v0.7.7 (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/251">#251</a>)
by <a href="https://github.com/prochac"><code>@prochac</code></a> in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/252">databricks/databricks-sql-go#252</a></li>
<li>fix(security): update dependencies to fix CVE by <a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/258">databricks/databricks-sql-go#258</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/prochac"><code>@prochac</code></a> made
their first contribution in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/252">databricks/databricks-sql-go#252</a></li>
<li><a
href="https://github.com/vikrantpuppala"><code>@vikrantpuppala</code></a>
made their first contribution in <a
href="https://redirect.github.com/databricks/databricks-sql-go/pull/258">databricks/databricks-sql-go#258</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/databricks/databricks-sql-go/compare/v1.6.1...v1.6.2">https://github.com/databricks/databricks-sql-go/compare/v1.6.1...v1.6.2</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sql-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sql-go's
changelog</a>.</em></p>
<blockquote>
<h2>v1.8.0 (2025-07-24)</h2>
<ul>
<li>Add Arrow IPC Iterator</li>
</ul>
<h2>v1.7.1 (2025-05-20)</h2>
<ul>
<li><code>databricks/databricks-sql-go#267</code></li>
<li><code>databricks/databricks-sql-go#268</code></li>
<li><code>databricks/databricks-sql-go#266</code></li>
<li><code>databricks/databricks-sql-go#264</code></li>
</ul>
<h2>v1.7.0 (2025-04-09)</h2>
<ul>
<li><code>databricks/databricks-sql-go#260</code></li>
<li><code>databricks/databricks-sql-go#261</code></li>
</ul>
<h2>v1.6.2 (2025-03-18)</h2>
<ul>
<li><code>databricks/databricks-sql-go#247</code></li>
<li><code>databricks/databricks-sql-go#249</code></li>
<li><code>databricks/databricks-sql-go#251</code></li>
<li><code>databricks/databricks-sql-go#258</code></li>
<li><code>databricks/databricks-sql-go#255</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/529d69c5ce36b0eeaad9ec573cfd77a7342cc5ff"><code>529d69c</code></a>
Prepare for v1.8.0 release (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/280">#280</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/7bde0dc3d1a27a22354d50129a59e477af17add2"><code>7bde0dc</code></a>
Add Arrow IPC stream iterator for direct access to Arrow buffer (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/279">#279</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/746c05de3ec5c8f06603879998a5fb703a35d473"><code>746c05d</code></a>
Ensure thrift field IDs stay within range (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/276">#276</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/039c53ab5e21dbbfae5d29aa4595d36ea7a3465e"><code>039c53a</code></a>
Add test to ensure thrift field IDs stay within range</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/12d2ceda465a8296b2d2080550c91581c71b3605"><code>12d2ced</code></a>
Prepare for v1.7.1 (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/271">#271</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/01cd3d122125a0ec02aa035e62823fc6ef408911"><code>01cd3d1</code></a>
Update CODEOWNERS (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/272">#272</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/2361714c73a38a23aed7334bf8240238803b608d"><code>2361714</code></a>
Update CODEOWNERS</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/149e1ad615c847e33895c5d7299e6031cd593ad6"><code>149e1ad</code></a>
Prepare for v1.7.1</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/c29199287a059c1056d1179402088acdd30706e1"><code>c291992</code></a>
Add schema to ArrowBatchIterator (<a
href="https://redirect.github.com/databricks/databricks-sql-go/issues/267">#267</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sql-go/commit/7263c534751313073578e0adc3f0e34e15f75bad"><code>7263c53</code></a>
[PECOBLR-402] Update thrift client library after cleaning up unused
fields an...</li>
<li>Additional commits viewable in <a
href="https://github.com/databricks/databricks-sql-go/compare/v1.6.1...v1.8.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/databricks/databricks-sql-go&package-manager=go_modules&previous-version=1.6.1&new-version=1.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

> **Note**
> Automatic rebases have been disabled on this pull request as it has
been open for over 30 days.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

jackyhu-db jackyhu-db approved these changes

deeksha-db Awaiting requested review from deeksha-db

yunbodeng-db Awaiting requested review from yunbodeng-db

andrefurlan-db Awaiting requested review from andrefurlan-db

rcypher-databricks Awaiting requested review from rcypher-databricks

jadewang-db Awaiting requested review from jadewang-db

Labels

None yet