Skip to content

Commit a6600af

Browse files
vladimirg-dbcloud-fan
authored andcommitted
[SPARK-51357][SQL] Preserve plan change logging level for views
### What changes were proposed in this pull request? Preserve plan change logging level for views: - `spark.sql.planChangeLog.level` - `spark.sql.expressionTreeChangeLog.level` Before: ![image](https://github.com/user-attachments/assets/bf9f5804-1561-499a-9ba6-6d88e9b16aea) After: ![image](https://github.com/user-attachments/assets/f076784a-bae1-4626-8a46-a8e1b3078c01) ### Why are the changes needed? Plan change logging level is not preserved for views. This hides part of the resolution process for queries with views and makes debugging harder. ### Does this PR introduce _any_ user-facing change? No, just debugging becomes easier. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#50118 from vladimirg-db/vladimir-golubev_data/preserve-plan-change-logging-setting-for-views. Authored-by: Vladimir Golubev <vladimir.golubev@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 7fca7ae commit a6600af

File tree

1 file changed

+19
-9
lines changed

1 file changed

+19
-9
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -840,20 +840,30 @@ object View {
840840
// For temporary view, we always use captured sql configs
841841
if (activeConf.useCurrentSQLConfigsForView && !isTempView) return activeConf
842842

843-
val sqlConf = new SQLConf()
844843
// We retain below configs from current session because they are not captured by view
845844
// as optimization configs but they are still needed during the view resolution.
846-
// TODO: remove this `retainedConfigs` after the `RelationConversions` is moved to
845+
// TODO: remove this `retainedHiveConfigs` after the `RelationConversions` is moved to
847846
// optimization phase.
847+
val retainedHiveConfigs = Seq(
848+
"spark.sql.hive.convertMetastoreParquet",
849+
"spark.sql.hive.convertMetastoreOrc",
850+
"spark.sql.hive.convertInsertingPartitionedTable",
851+
"spark.sql.hive.convertInsertingUnpartitionedTable",
852+
"spark.sql.hive.convertMetastoreCtas"
853+
)
854+
855+
val retainedLoggingConfigs = Seq(
856+
"spark.sql.planChangeLog.level",
857+
"spark.sql.expressionTreeChangeLog.level"
858+
)
859+
848860
val retainedConfigs = activeConf.getAllConfs.filter { case (key, _) =>
849-
Seq(
850-
"spark.sql.hive.convertMetastoreParquet",
851-
"spark.sql.hive.convertMetastoreOrc",
852-
"spark.sql.hive.convertInsertingPartitionedTable",
853-
"spark.sql.hive.convertInsertingUnpartitionedTable",
854-
"spark.sql.hive.convertMetastoreCtas"
855-
).contains(key) || key.startsWith("spark.sql.catalog.")
861+
retainedHiveConfigs.contains(key) || retainedLoggingConfigs.contains(key) || key.startsWith(
862+
"spark.sql.catalog."
863+
)
856864
}
865+
866+
val sqlConf = new SQLConf()
857867
for ((k, v) <- configs ++ retainedConfigs) {
858868
sqlConf.settings.put(k, v)
859869
}

0 commit comments

Comments
 (0)