Skip to content

Commit f65dc95

Browse files
LinhongLiucloud-fan
authored andcommitted
[SPARK-26526][SQL][TEST] Fix invalid test case about non-deterministic expression
## What changes were proposed in this pull request? Test case in SPARK-10316 is used to make sure non-deterministic `Filter` won't be pushed through `Project` But in current code base this test case can't cover this purpose. Change LogicalRDD to HadoopFsRelation can fix this issue. ## How was this patch tested? Modified test pass. Closes #23440 from LinhongLiu/fix-test. Authored-by: Liu,Linhong <liulinhong@baidu.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 05372d1 commit f65dc95

File tree

1 file changed

+7
-4
lines changed

1 file changed

+7
-4
lines changed

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1398,11 +1398,14 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
13981398
}
13991399

14001400
test("SPARK-10316: respect non-deterministic expressions in PhysicalOperation") {
1401-
val input = spark.read.json((1 to 10).map(i => s"""{"id": $i}""").toDS())
1401+
withTempDir { dir =>
1402+
(1 to 10).toDF("id").write.mode(SaveMode.Overwrite).json(dir.getCanonicalPath)
1403+
val input = spark.read.json(dir.getCanonicalPath)
14021404

1403-
val df = input.select($"id", rand(0).as('r))
1404-
df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === $"b.id").collect().foreach { row =>
1405-
assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
1405+
val df = input.select($"id", rand(0).as('r))
1406+
df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === $"b.id").collect().foreach { row =>
1407+
assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
1408+
}
14061409
}
14071410
}
14081411

0 commit comments

Comments
 (0)