Skip to content

[SPARK-29947][SQL] Improve ResolveRelations performance #26589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

[SPARK-29947][SQL] Improve ResolveRelations performance #26589

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3379cf3
Improve ResolveTables and ResolveRelations performance
wangyum Nov 19, 2019
a7ce2a7
Merge remote-tracking branch 'upstream/master' into SPARK-29947
wangyum Nov 25, 2019
84967f5
Merge remote-tracking branch 'upstream/master' into SPARK-29947
wangyum Dec 15, 2019
100ea1a
Merge remote-tracking branch 'upstream/master' into SPARK-29947
wangyum Dec 27, 2019
07e6b7f
Add map to AnalysisContext
wangyum Dec 28, 2019
77b739a
Fix test error
wangyum Dec 29, 2019
2e1f87d
Change relationCache key from UnresolvedRelation to String
wangyum Jan 2, 2020
e88351a
Merge remote-tracking branch 'upstream/master' into SPARK-29947
wangyum Jan 2, 2020
99a8557
Apply the cache in lookupRelation
wangyum Jan 3, 2020
8e7b666
v1Table.v1Table.qualifiedName -> catalog.name +: ident.namespace :+ i…
wangyum Jan 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,14 @@ object FakeV2SessionCatalog extends TableCatalog {
* views.
* @param nestedViewDepth The nested depth in the view resolution, this enables us to limit the
* depth of nested views.
* @param relationCache A mapping from qualified table names to resolved relations. This can ensure
* that the table is resolved only once if a table is used multiple times
* in a query.
*/
case class AnalysisContext(
catalogAndNamespace: Seq[String] = Nil,
nestedViewDepth: Int = 0)
nestedViewDepth: Int = 0,
relationCache: mutable.Map[Seq[String], LogicalPlan] = mutable.Map.empty)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the anti-pattern from the style guide ... https://github.com/databricks/scala-style-guide#case-classes-and-immutability


object AnalysisContext {
private val value = new ThreadLocal[AnalysisContext]() {
Expand All @@ -110,7 +114,7 @@ object AnalysisContext {
def withAnalysisContext[A](catalogAndNamespace: Seq[String])(f: => A): A = {
val originContext = value.get()
val context = AnalysisContext(
catalogAndNamespace, nestedViewDepth = originContext.nestedViewDepth + 1)
catalogAndNamespace, originContext.nestedViewDepth + 1, originContext.relationCache)
set(context)
try f finally { set(originContext) }
}
Expand Down Expand Up @@ -870,7 +874,9 @@ class Analyzer(
case SessionCatalogAndIdentifier(catalog, ident) =>
CatalogV2Util.loadTable(catalog, ident).map {
case v1Table: V1Table =>
v1SessionCatalog.getRelation(v1Table.v1Table)
val key = catalog.name +: ident.namespace :+ ident.name
AnalysisContext.get.relationCache.getOrElseUpdate(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work. The table is already loaded and it's too late to use the cache. I've sent #27341 to fix it.

key, v1SessionCatalog.getRelation(v1Table.v1Table))
case table =>
DataSourceV2Relation.create(table)
}
Expand Down