Skip to content

Commit 14b82dd

Browse files
yaooqinndongjoon-hyun
authored andcommitted
[SPARK-47938][SQL] MsSQLServer: Cannot find data type BYTE error
### What changes were proposed in this pull request? This PR uses SMALLINT (as TINYINT ranges [0, 255]) instead of BYTE to fix the ByteType mapping for MsSQLServer JDBC ```java [info] com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable apache#1: Cannot find data type BYTE. [info] at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1662) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:898) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:793) [info] at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7417) [info] at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3488) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:262) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:237) [info] at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeUpdate(SQLServerStatement.java:733) [info] at org.apache.spark.sql.jdbc.JdbcDialect.createTable(JdbcDialects.scala:267) ``` ### Why are the changes needed? bugfix ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#46164 from yaooqinn/SPARK-47938. Lead-authored-by: Kent Yao <yao@apache.org> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 1c1a74f commit 14b82dd

File tree

2 files changed

+9
-0
lines changed

2 files changed

+9
-0
lines changed

connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -437,4 +437,12 @@ class MsSqlServerIntegrationSuite extends DockerJDBCIntegrationSuite {
437437
.load()
438438
assert(df.collect().toSet === expectedResult)
439439
}
440+
441+
test("SPARK-47938: Fix 'Cannot find data type BYTE' in SQL Server") {
442+
spark.sql("select cast(1 as byte) as c0")
443+
.write
444+
.jdbc(jdbcUrl, "test_byte", new Properties)
445+
val df = spark.read.jdbc(jdbcUrl, "test_byte", new Properties)
446+
checkAnswer(df, Row(1.toShort))
447+
}
440448
}

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,7 @@ private case class MsSqlServerDialect() extends JdbcDialect {
136136
case BinaryType => Some(JdbcType("VARBINARY(MAX)", java.sql.Types.VARBINARY))
137137
case ShortType if !SQLConf.get.legacyMsSqlServerNumericMappingEnabled =>
138138
Some(JdbcType("SMALLINT", java.sql.Types.SMALLINT))
139+
case ByteType => Some(JdbcType("SMALLINT", java.sql.Types.TINYINT))
139140
case _ => None
140141
}
141142

0 commit comments

Comments
 (0)