[SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs

srowen · srowen · commit 51ade51a9fd6 · 2016-07-13T11:39:32.000+01:00
## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable resource cleanup ## How was this patch tested? Jenkins tests Author: Sean Owen <sowen@cloudera.com> Closes apache#14153 from srowen/SPARK-16440.
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
@@ -434,6 +434,9 @@ class Word2Vec extends Serializable with Logging {
       bcSyn1Global.unpersist(false)
     }
     newSentences.unpersist()
+    expTable.unpersist()
+    bcVocab.unpersist()
+    bcVocabHash.unpersist()
 
     val wordArray = vocab.map(_.word)
     new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)

Original file line number	Diff line number	Diff line change
`@@ -434,6 +434,9 @@ class Word2Vec extends Serializable with Logging {`
`434`	`434`	`bcSyn1Global.unpersist(false)`
`435`	`435`	`}`
`436`	`436`	`newSentences.unpersist()`
	`437`	`+ expTable.unpersist()`
	`438`	`+ bcVocab.unpersist()`
	`439`	`+ bcVocabHash.unpersist()`
`437`	`440`
`438`	`441`	`val wordArray = vocab.map(_.word)`
`439`	`442`	`new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)`