Skip to content

Commit ea37717

Browse files
maropuHyukjinKwon
authored andcommitted
[SPARK-32106][SQL][FOLLOWUP] Fix flaky tests in transform.sql
### What changes were proposed in this pull request? This PR intends to fix flaky GitHub Actions (GA) tests below in `transform.sql` (this flakiness does not seem to happen in the Jenkins tests): - https://github.com/apache/spark/runs/1592987501 - https://github.com/apache/spark/runs/1593196242 - https://github.com/apache/spark/runs/1595496305 - https://github.com/apache/spark/runs/1596309555 This is because the error message is different between test runs in GA (the error message seems to be truncated indeterministically) ,e.g., ``` # https://github.com/apache/spark/runs/1592987501 Expected "...h status 127. Error:[ /bin/bash: some_non_existent_command: command not found]", but got "...h status 127. Error:[]" Result did not match for query #2 # https://github.com/apache/spark/runs/1593196242 Expected "...istent_command: comm[and not found]", but got "...istent_command: comm[]" Result did not match for query #2 ``` The root cause of this indeterministic behaviour happening only in GA is not clear though, this test throws SparkException consistently even in GA. So, this PR proposes to make the test just check if it will be thrown when running it. This PR comes from the dongjoon-hyun comment: https://github.com/apache/spark/pull/29414/files#r547414513 ### Why are the changes needed? Bugfix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added tests. Closes apache#30896 from maropu/SPARK-32106-FOLLOWUP. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent a3dd8da commit ea37717

File tree

3 files changed

+21
-33
lines changed

3 files changed

+21
-33
lines changed

sql/core/src/test/resources/sql-tests/inputs/transform.sql

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,6 @@ SELECT TRANSFORM(a)
99
USING 'cat' AS (a)
1010
FROM t;
1111

12-
-- with non-exist command
13-
SELECT TRANSFORM(a)
14-
USING 'some_non_existent_command' AS (a)
15-
FROM t;
16-
17-
-- with non-exist file
18-
SELECT TRANSFORM(a)
19-
USING 'python some_non_existent_file' AS (a)
20-
FROM t;
21-
2212
-- common supported data types between no serde and serde transform
2313
SELECT a, b, decode(c, 'UTF-8'), d, e, f, g, h, i, j, k, l FROM (
2414
SELECT TRANSFORM(a, b, c, d, e, f, g, h, i, j, k, l)

sql/core/src/test/resources/sql-tests/results/transform.sql.out

Lines changed: 1 addition & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- Automatically generated by SQLQueryTestSuite
2-
-- Number of queries: 18
2+
-- Number of queries: 16
33

44

55
-- !query
@@ -26,28 +26,6 @@ struct<a:string>
2626
3
2727

2828

29-
-- !query
30-
SELECT TRANSFORM(a)
31-
USING 'some_non_existent_command' AS (a)
32-
FROM t
33-
-- !query schema
34-
struct<>
35-
-- !query output
36-
org.apache.spark.SparkException
37-
Subprocess exited with status 127. Error: /bin/bash: some_non_existent_command: command not found
38-
39-
40-
-- !query
41-
SELECT TRANSFORM(a)
42-
USING 'python some_non_existent_file' AS (a)
43-
FROM t
44-
-- !query schema
45-
struct<>
46-
-- !query output
47-
org.apache.spark.SparkException
48-
Subprocess exited with status 2. Error: python: can't open file 'some_non_existent_file': [Errno 2] No such file or directory
49-
50-
5129
-- !query
5230
SELECT a, b, decode(c, 'UTF-8'), d, e, f, g, h, i, j, k, l FROM (
5331
SELECT TRANSFORM(a, b, c, d, e, f, g, h, i, j, k, l)

sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -420,6 +420,26 @@ abstract class BaseScriptTransformationSuite extends SparkPlanTest with SQLTestU
420420
'b.cast("string").as("b"),
421421
lit(null), lit(null)).collect())
422422
}
423+
424+
test("SPARK-32106: TRANSFORM with non-existent command/file") {
425+
Seq(
426+
s"""
427+
|SELECT TRANSFORM(a)
428+
|USING 'some_non_existent_command' AS (a)
429+
|FROM VALUES (1) t(a)
430+
""".stripMargin,
431+
s"""
432+
|SELECT TRANSFORM(a)
433+
|USING 'python some_non_existent_file' AS (a)
434+
|FROM VALUES (1) t(a)
435+
""".stripMargin).foreach { query =>
436+
intercept[SparkException] {
437+
// Since an error message is shell-dependent, this test just checks
438+
// if the expected exception will be thrown.
439+
sql(query).collect()
440+
}
441+
}
442+
}
423443
}
424444

425445
case class ExceptionInjectingOperator(child: SparkPlan) extends UnaryExecNode {

0 commit comments

Comments
 (0)