JDBC driver conditional sets fetchSize on opening session #6036

pan3793 · 2024-01-31T12:10:19Z

🔍 Description

Issue References 🔗

I get reported by a user that using Kyuubi JDBC driver 1.8.0 to access Spark Thrift Server 2.4 (with Hive 1.2.1) with getting an error on opening the session even with clientProtocol=7

org.apache.hive.service.cli.HiveSQLException: java.lang.IllegalArgumentException: hive configuration hive.server2.thrift.resultset.default.fetch.size does not exists.
        at org.apache.hive.service.cli.session.HiveSessionImpl.configureSession(HiveSessionImpl.java:220)
        at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:154)
        at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:258)
        ... 13 more

Describe Your Solution 🔧

When hive.conf.validation is true (it also is the default value), an IllegalArgumentException will be thrown if the provided configurations start with hive. and can not be recognized by the server.

One solution is to disable hive.conf.validation on the server side, but we can also address it by avoiding passing this configuration if we know that the server does not support it.

HIVE-14901 (2.3.0, HIVE_CLI_SERVICE_PROTOCOL_V10) introduces hive.server2.thrift.resultset.default.fetch.size, so we can set this configuration only when protocol >= HIVE_CLI_SERVICE_PROTOCOL_V10

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Verified locally by connecting to HS2 2.1.1 and HS2 2.3.9

Behavior Without This Pull Request ⚰️

IllegalArgumentException throws when using jdbc:hive2://localhost:10000/default;clientProtocol=8 to connect HS2 2.1.1

Everything is OK when using jdbc:hive2://localhost:10000/default to connect HS2 2.3.9

Behavior With This Pull Request 🎉

Everything is OK when using jdbc:hive2://localhost:10000/default;clientProtocol=8 to connect HS2 2.1.1

Everything is OK when using jdbc:hive2://localhost:10000/default to connect HS2 2.3.9

Checklist 📝

This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

YuxiangZhuang

look good to me

# 🔍 Description ## Issue References 🔗 I get reported by a user that using Kyuubi JDBC driver 1.8.0 to access Spark Thrift Server 2.4 (with Hive 1.2.1) with getting an error on opening the session even with `clientProtocol=7` ``` org.apache.hive.service.cli.HiveSQLException: java.lang.IllegalArgumentException: hive configuration hive.server2.thrift.resultset.default.fetch.size does not exists. at org.apache.hive.service.cli.session.HiveSessionImpl.configureSession(HiveSessionImpl.java:220) at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:154) at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:258) ... 13 more ``` ## Describe Your Solution 🔧 When `hive.conf.validation` is `true` (it also is the default value), an IllegalArgumentException will be thrown if the provided configurations start with `hive.` and can not be recognized by the server. One solution is to disable `hive.conf.validation` on the server side, but we can also address it by avoiding passing this configuration if we know that the server does not support it. HIVE-14901 (2.3.0, HIVE_CLI_SERVICE_PROTOCOL_V10) introduces `hive.server2.thrift.resultset.default.fetch.size`, so we can set this configuration only when protocol >= HIVE_CLI_SERVICE_PROTOCOL_V10 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Verified locally by connecting to HS2 2.1.1 and HS2 2.3.9 #### Behavior Without This Pull Request ⚰️ IllegalArgumentException throws when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 #### Behavior With This Pull Request 🎉 Everything is OK when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6036 from pan3793/jdbc-fetchsize. Closes #6036 7ea91f1 [Cheng Pan] nit e6ea829 [Cheng Pan] fix 1dbecbb [Cheng Pan] JDBC driver conditional sets fetchSize on opening session Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 4bd259a) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2024-01-31T13:10:10Z

Thanks, merged to master/1.8

…g session # 🔍 Description ## Issue References 🔗 I get reported by a user that using Kyuubi JDBC driver 1.8.0 to access Spark Thrift Server 2.4 (with Hive 1.2.1) with getting an error on opening the session even with `clientProtocol=7` ``` org.apache.hive.service.cli.HiveSQLException: java.lang.IllegalArgumentException: hive configuration hive.server2.thrift.resultset.default.fetch.size does not exists. at org.apache.hive.service.cli.session.HiveSessionImpl.configureSession(HiveSessionImpl.java:220) at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:154) at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:258) ... 13 more ``` ## Describe Your Solution 🔧 When `hive.conf.validation` is `true` (it also is the default value), an IllegalArgumentException will be thrown if the provided configurations start with `hive.` and can not be recognized by the server. One solution is to disable `hive.conf.validation` on the server side, but we can also address it by avoiding passing this configuration if we know that the server does not support it. HIVE-14901 (2.3.0, HIVE_CLI_SERVICE_PROTOCOL_V10) introduces `hive.server2.thrift.resultset.default.fetch.size`, so we can set this configuration only when protocol >= HIVE_CLI_SERVICE_PROTOCOL_V10 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Verified locally by connecting to HS2 2.1.1 and HS2 2.3.9 #### Behavior Without This Pull Request ⚰️ IllegalArgumentException throws when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 #### Behavior With This Pull Request 🎉 Everything is OK when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6036 from pan3793/jdbc-fetchsize. Closes apache#6036 7ea91f1 [Cheng Pan] nit e6ea829 [Cheng Pan] fix 1dbecbb [Cheng Pan] JDBC driver conditional sets fetchSize on opening session Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

…se notes # 🔍 Description ## Issue References 🔗 Currently, we use a rather primitive way to manually write release notes from scratch, and some of the mechanical and repetitive work can be simplified by the scripts. ## Describe Your Solution 🔧 Adds a script to simplify the process of creating release notes. Note: it just simplifies some processes, the release manager still needs to tune the outputs by hand. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` RELEASE_TAG=v1.8.1 PREVIOUS_RELEASE_TAG=v1.8.0 build/release/pre_gen_release_notes.py ``` ``` $ head build/release/commits-v1.8.1.txt [KYUUBI #5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central [KYUUBI #6058] Make Jetty server stop timeout configurable [KYUUBI #5952][1.8] Disconnect connections without running operations after engine maxlife time graceful period [KYUUBI #6048] Assign serviceNode and add volatile for variables [KYUUBI #5991] Error on reading Atlas properties composed of multi values [KYUUBI #6045] [REST] Sync the AdminRestApi with the AdminResource Apis [KYUUBI #6047] [CI] Free up disk space [KYUUBI #6036] JDBC driver conditional sets fetchSize on opening session [KYUUBI #6028] Exited spark-submit process should not block batch submit queue [KYUUBI #6018] Speed up GetTables operation for Spark session catalog ``` ``` $ head build/release/contributors-v1.8.1.txt * Shaoyun Chen -- [KYUUBI #5857][KYUUBI #5720][KYUUBI #5785][KYUUBI #5617] * Chao Chen -- [KYUUBI #5750] * Flyangz -- [KYUUBI #5832] * Pengqi Li -- [KYUUBI #5713] * Bowen Liang -- [KYUUBI #5730][KYUUBI #5802][KYUUBI #5767][KYUUBI #5831][KYUUBI #5801][KYUUBI #5754][KYUUBI #5626][KYUUBI #5811][KYUUBI #5853][KYUUBI #5765] * Paul Lin -- [KYUUBI #5799][KYUUBI #5814] * Senmiao Liu -- [KYUUBI #5969][KYUUBI #5244] * Xiao Liu -- [KYUUBI #5962] * Peiyue Liu -- [KYUUBI #5331] * Junjie Ma -- [KYUUBI #5789] ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6074 from pan3793/release-script. Closes #6074 3d5ec20 [Cheng Pan] credits 1765279 [Cheng Pan] Add a script to simplify the process of creating release notes Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

…g session # 🔍 Description ## Issue References 🔗 I get reported by a user that using Kyuubi JDBC driver 1.8.0 to access Spark Thrift Server 2.4 (with Hive 1.2.1) with getting an error on opening the session even with `clientProtocol=7` ``` org.apache.hive.service.cli.HiveSQLException: java.lang.IllegalArgumentException: hive configuration hive.server2.thrift.resultset.default.fetch.size does not exists. at org.apache.hive.service.cli.session.HiveSessionImpl.configureSession(HiveSessionImpl.java:220) at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:154) at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:258) ... 13 more ``` ## Describe Your Solution 🔧 When `hive.conf.validation` is `true` (it also is the default value), an IllegalArgumentException will be thrown if the provided configurations start with `hive.` and can not be recognized by the server. One solution is to disable `hive.conf.validation` on the server side, but we can also address it by avoiding passing this configuration if we know that the server does not support it. HIVE-14901 (2.3.0, HIVE_CLI_SERVICE_PROTOCOL_V10) introduces `hive.server2.thrift.resultset.default.fetch.size`, so we can set this configuration only when protocol >= HIVE_CLI_SERVICE_PROTOCOL_V10 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Verified locally by connecting to HS2 2.1.1 and HS2 2.3.9 #### Behavior Without This Pull Request ⚰️ IllegalArgumentException throws when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 #### Behavior With This Pull Request 🎉 Everything is OK when using `jdbc:hive2://localhost:10000/default;clientProtocol=8` to connect HS2 2.1.1 Everything is OK when using `jdbc:hive2://localhost:10000/default` to connect HS2 2.3.9 --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6036 from pan3793/jdbc-fetchsize. Closes apache#6036 7ea91f1 [Cheng Pan] nit e6ea829 [Cheng Pan] fix 1dbecbb [Cheng Pan] JDBC driver conditional sets fetchSize on opening session Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

… release notes # 🔍 Description ## Issue References 🔗 Currently, we use a rather primitive way to manually write release notes from scratch, and some of the mechanical and repetitive work can be simplified by the scripts. ## Describe Your Solution 🔧 Adds a script to simplify the process of creating release notes. Note: it just simplifies some processes, the release manager still needs to tune the outputs by hand. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` RELEASE_TAG=v1.8.1 PREVIOUS_RELEASE_TAG=v1.8.0 build/release/pre_gen_release_notes.py ``` ``` $ head build/release/commits-v1.8.1.txt [KYUUBI apache#5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central [KYUUBI apache#6058] Make Jetty server stop timeout configurable [KYUUBI apache#5952][1.8] Disconnect connections without running operations after engine maxlife time graceful period [KYUUBI apache#6048] Assign serviceNode and add volatile for variables [KYUUBI apache#5991] Error on reading Atlas properties composed of multi values [KYUUBI apache#6045] [REST] Sync the AdminRestApi with the AdminResource Apis [KYUUBI apache#6047] [CI] Free up disk space [KYUUBI apache#6036] JDBC driver conditional sets fetchSize on opening session [KYUUBI apache#6028] Exited spark-submit process should not block batch submit queue [KYUUBI apache#6018] Speed up GetTables operation for Spark session catalog ``` ``` $ head build/release/contributors-v1.8.1.txt * Shaoyun Chen -- [KYUUBI apache#5857][KYUUBI apache#5720][KYUUBI apache#5785][KYUUBI apache#5617] * Chao Chen -- [KYUUBI apache#5750] * Flyangz -- [KYUUBI apache#5832] * Pengqi Li -- [KYUUBI apache#5713] * Bowen Liang -- [KYUUBI apache#5730][KYUUBI apache#5802][KYUUBI apache#5767][KYUUBI apache#5831][KYUUBI apache#5801][KYUUBI apache#5754][KYUUBI apache#5626][KYUUBI apache#5811][KYUUBI apache#5853][KYUUBI apache#5765] * Paul Lin -- [KYUUBI apache#5799][KYUUBI apache#5814] * Senmiao Liu -- [KYUUBI apache#5969][KYUUBI apache#5244] * Xiao Liu -- [KYUUBI apache#5962] * Peiyue Liu -- [KYUUBI apache#5331] * Junjie Ma -- [KYUUBI apache#5789] ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6074 from pan3793/release-script. Closes apache#6074 3d5ec20 [Cheng Pan] credits 1765279 [Cheng Pan] Add a script to simplify the process of creating release notes Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

JDBC driver conditional sets fetchSize on opening session

1dbecbb

github-actions bot added module:hive module:ctl labels Jan 31, 2024

pan3793 added 2 commits January 31, 2024 20:33

fix

e6ea829

nit

7ea91f1

YuxiangZhuang approved these changes Jan 31, 2024

View reviewed changes

pan3793 self-assigned this Jan 31, 2024

pan3793 added this to the v1.8.1 milestone Jan 31, 2024

pan3793 requested a review from cxzl25 January 31, 2024 12:51

cxzl25 approved these changes Jan 31, 2024

View reviewed changes

pan3793 closed this in 4bd259a Jan 31, 2024

pan3793 deleted the jdbc-fetchsize branch February 1, 2024 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDBC driver conditional sets fetchSize on opening session #6036

JDBC driver conditional sets fetchSize on opening session #6036

pan3793 commented Jan 31, 2024 •

edited

Loading

YuxiangZhuang left a comment

pan3793 commented Jan 31, 2024

JDBC driver conditional sets fetchSize on opening session #6036

JDBC driver conditional sets fetchSize on opening session #6036

Conversation

pan3793 commented Jan 31, 2024 • edited Loading

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Checklist 📝

YuxiangZhuang left a comment

Choose a reason for hiding this comment

pan3793 commented Jan 31, 2024

pan3793 commented Jan 31, 2024 •

edited

Loading