Skip to content

Commit 5a2f374

Browse files
n-young-dbJoshRosen
authored andcommitted
[SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
### What changes were proposed in this pull request? - Changed the `ineffectiveRules` variable of the `TreeNode` class to initialize lazily. This will reduce unnecessary driver memory pressure. ### Why are the changes needed? - Plans with large expression or operator trees are known to cause driver memory pressure; this is one step in alleviating that issue. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UT covers behavior. Outwards facing behavior does not change. ### Was this patch authored or co-authored using generative AI tooling? No Closes #46919 from n-young-db/ineffective-rules-lazy. Authored-by: Nick Young <nick.young@databricks.com> Signed-off-by: Josh Rosen <joshrosen@databricks.com>
1 parent ec6db63 commit 5a2f374

File tree

1 file changed

+9
-2
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees

1 file changed

+9
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,14 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
120120
* ineffective for subsequent apply calls on this tree because query plan structures are
121121
* immutable.
122122
*/
123-
private val ineffectiveRules: BitSet = new BitSet(RuleIdCollection.NumRules)
123+
private[this] var _ineffectiveRules: BitSet = null
124+
private def ineffectiveRules: BitSet = {
125+
if (_ineffectiveRules eq null) {
126+
_ineffectiveRules = new BitSet(RuleIdCollection.NumRules)
127+
}
128+
_ineffectiveRules
129+
}
130+
private def isIneffectiveRulesEmpty = _ineffectiveRules eq null
124131

125132
/**
126133
* @return a sequence of tree pattern enums in a TreeNode T. It does not include propagated
@@ -149,7 +156,7 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
149156
* UnknownId, it returns false.
150157
*/
151158
protected def isRuleIneffective(ruleId : RuleId): Boolean = {
152-
if (ruleId eq UnknownRuleId) {
159+
if (isIneffectiveRulesEmpty || (ruleId eq UnknownRuleId)) {
153160
return false
154161
}
155162
ineffectiveRules.get(ruleId.id)

0 commit comments

Comments
 (0)