[native_datafusion] Subquery has not finished #2672
Description
Describe the bug
When running CometPlanStabilitySuite with native_datafusion scan, I see query failures such as:
- check simplified (tpcds-v1.4/q6) - native_datafusion *** FAILED *** (122 milliseconds)
java.lang.IllegalArgumentException: requirement failed: Subquery scalar-subquery#82944, [id=#198734] has not finished
at scala.Predef$.require(Predef.scala:337)
at org.apache.spark.sql.execution.ScalarSubquery.toLiteral(subquery.scala:108)
at org.apache.spark.sql.execution.FileSourceScanLike.$anonfun$pushedDownFilters$1(DataSourceScanExec.scala:426)
at org.apache.spark.sql.execution.FileSourceScanLike$$anonfun$$nestedInanonfun$translateToV1Filters$1$1.applyOrElse(DataSourceScanExec.scala:410)
at org.apache.spark.sql.execution.FileSourceScanLike$$anonfun$$nestedInanonfun$translateToV1Filters$1$1.applyOrElse(DataSourceScanExec.scala:407)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:470)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:86)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:470)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:475)
at org.apache.spark.sql.catalyst.trees.BinaryLike.mapChildren(TreeNode.scala:1259)
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response