Skip to content

Commit 96f65c9

Browse files
JoshRosendongjoon-hyun
authored andcommitted
[SPARK-48128][SQL] For BitwiseCount / bit_count expression, fix codegen syntax error for boolean type inputs
### What changes were proposed in this pull request? This PR fixes an issue where `BitwiseCount` / `bit_count` of boolean inputs would cause codegen to generate syntactically invalid Java code that does not compile, triggering errors like ``` java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41, Column 11: Failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41, Column 11: Unexpected token "if" in primary ``` Even though this code has test cases in `bitwise.sql` via the query test framework, those existing test cases were insufficient to find this problem: I believe that is because the example queries were constant-folded using the interpreted path, leaving the codegen path without test coverage. This PR fixes the codegen issue and adds explicit expression tests to ensure that the same tests run on both the codegen and interpreted paths. ### Why are the changes needed? Fix a rare codegen to interpreted fallback issue, which may harm query performance. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new test cases to BitwiseExpressionsSuite.scala, copied from the existing `bitwise.sql` query test case file. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46382 from JoshRosen/SPARK-48128-bit_count_codegen. Authored-by: Josh Rosen <joshrosen@databricks.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent ca8c269 commit 96f65c9

File tree

2 files changed

+42
-1
lines changed

2 files changed

+42
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ case class BitwiseCount(child: Expression)
229229
override def prettyName: String = "bit_count"
230230

231231
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = child.dataType match {
232-
case BooleanType => defineCodeGen(ctx, ev, c => s"if ($c) 1 else 0")
232+
case BooleanType => defineCodeGen(ctx, ev, c => s"($c) ? 1 : 0")
233233
case _ => defineCodeGen(ctx, ev, c => s"java.lang.Long.bitCount($c)")
234234
}
235235

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/BitwiseExpressionsSuite.scala

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,47 @@ class BitwiseExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
134134
}
135135
}
136136

137+
test("BitCount") {
138+
// null
139+
val nullLongLiteral = Literal.create(null, LongType)
140+
val nullIntLiteral = Literal.create(null, IntegerType)
141+
val nullBooleanLiteral = Literal.create(null, BooleanType)
142+
checkEvaluation(BitwiseCount(nullLongLiteral), null)
143+
checkEvaluation(BitwiseCount(nullIntLiteral), null)
144+
checkEvaluation(BitwiseCount(nullBooleanLiteral), null)
145+
146+
// boolean
147+
checkEvaluation(BitwiseCount(Literal(true)), 1)
148+
checkEvaluation(BitwiseCount(Literal(false)), 0)
149+
150+
// byte/tinyint
151+
checkEvaluation(BitwiseCount(Literal(1.toByte)), 1)
152+
checkEvaluation(BitwiseCount(Literal(2.toByte)), 1)
153+
checkEvaluation(BitwiseCount(Literal(3.toByte)), 2)
154+
155+
// short/smallint
156+
checkEvaluation(BitwiseCount(Literal(1.toShort)), 1)
157+
checkEvaluation(BitwiseCount(Literal(2.toShort)), 1)
158+
checkEvaluation(BitwiseCount(Literal(3.toShort)), 2)
159+
160+
// int
161+
checkEvaluation(BitwiseCount(Literal(1)), 1)
162+
checkEvaluation(BitwiseCount(Literal(2)), 1)
163+
checkEvaluation(BitwiseCount(Literal(3)), 2)
164+
165+
// long/bigint
166+
checkEvaluation(BitwiseCount(Literal(1L)), 1)
167+
checkEvaluation(BitwiseCount(Literal(2L)), 1)
168+
checkEvaluation(BitwiseCount(Literal(3L)), 2)
169+
170+
// negative num
171+
checkEvaluation(BitwiseCount(Literal(-1L)), 64)
172+
173+
// edge value
174+
checkEvaluation(BitwiseCount(Literal(9223372036854775807L)), 63)
175+
checkEvaluation(BitwiseCount(Literal(-9223372036854775808L)), 1)
176+
}
177+
137178
test("BitGet") {
138179
val nullLongLiteral = Literal.create(null, LongType)
139180
val nullIntLiteral = Literal.create(null, IntegerType)

0 commit comments

Comments
 (0)