Skip to content

Commit

Permalink
[SPARK-32106][SQL][FOLLOWUP] Fix flaky tests in transform.sql
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR intends to fix flaky GitHub Actions (GA) tests below in `transform.sql` (this flakiness does not seem to happen in the Jenkins tests):
- https://github.com/apache/spark/runs/1592987501
- https://github.com/apache/spark/runs/1593196242
- https://github.com/apache/spark/runs/1595496305
- https://github.com/apache/spark/runs/1596309555

This is because the error message is different between test runs in GA (the error message seems to be truncated indeterministically) ,e.g.,
```
# https://github.com/apache/spark/runs/1592987501
Expected "...h status 127. Error:[ /bin/bash: some_non_existent_command: command not found]", but got "...h status 127. Error:[]" Result did not match for query #2

# https://github.com/apache/spark/runs/1593196242
Expected "...istent_command: comm[and not found]", but got "...istent_command: comm[]" Result did not match for query #2
```
The root cause of this indeterministic behaviour happening only in GA is not clear though, this test throws SparkException consistently even in GA. So, this PR proposes to make the test just check if it will be thrown when running it.

This PR comes from the dongjoon-hyun comment: https://github.com/apache/spark/pull/29414/files#r547414513

### Why are the changes needed?

Bugfix.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Added tests.

Closes apache#30896 from maropu/SPARK-32106-FOLLOWUP.

Authored-by: Takeshi Yamamuro <yamamuro@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
maropu authored and HyukjinKwon committed Dec 23, 2020
1 parent a3dd8da commit ea37717
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 33 deletions.
10 changes: 0 additions & 10 deletions sql/core/src/test/resources/sql-tests/inputs/transform.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,6 @@ SELECT TRANSFORM(a)
USING 'cat' AS (a)
FROM t;

-- with non-exist command
SELECT TRANSFORM(a)
USING 'some_non_existent_command' AS (a)
FROM t;

-- with non-exist file
SELECT TRANSFORM(a)
USING 'python some_non_existent_file' AS (a)
FROM t;

-- common supported data types between no serde and serde transform
SELECT a, b, decode(c, 'UTF-8'), d, e, f, g, h, i, j, k, l FROM (
SELECT TRANSFORM(a, b, c, d, e, f, g, h, i, j, k, l)
Expand Down
24 changes: 1 addition & 23 deletions sql/core/src/test/resources/sql-tests/results/transform.sql.out
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 18
-- Number of queries: 16


-- !query
Expand All @@ -26,28 +26,6 @@ struct<a:string>
3


-- !query
SELECT TRANSFORM(a)
USING 'some_non_existent_command' AS (a)
FROM t
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkException
Subprocess exited with status 127. Error: /bin/bash: some_non_existent_command: command not found


-- !query
SELECT TRANSFORM(a)
USING 'python some_non_existent_file' AS (a)
FROM t
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkException
Subprocess exited with status 2. Error: python: can't open file 'some_non_existent_file': [Errno 2] No such file or directory


-- !query
SELECT a, b, decode(c, 'UTF-8'), d, e, f, g, h, i, j, k, l FROM (
SELECT TRANSFORM(a, b, c, d, e, f, g, h, i, j, k, l)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,26 @@ abstract class BaseScriptTransformationSuite extends SparkPlanTest with SQLTestU
'b.cast("string").as("b"),
lit(null), lit(null)).collect())
}

test("SPARK-32106: TRANSFORM with non-existent command/file") {
Seq(
s"""
|SELECT TRANSFORM(a)
|USING 'some_non_existent_command' AS (a)
|FROM VALUES (1) t(a)
""".stripMargin,
s"""
|SELECT TRANSFORM(a)
|USING 'python some_non_existent_file' AS (a)
|FROM VALUES (1) t(a)
""".stripMargin).foreach { query =>
intercept[SparkException] {
// Since an error message is shell-dependent, this test just checks
// if the expected exception will be thrown.
sql(query).collect()
}
}
}
}

case class ExceptionInjectingOperator(child: SparkPlan) extends UnaryExecNode {
Expand Down

0 comments on commit ea37717

Please sign in to comment.