[SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs

srowen · srowen · commit 4381e2121401 · 2016-07-13T11:39:49.000+01:00
## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable resource cleanup ## How was this patch tested? Jenkins tests Author: Sean Owen <sowen@cloudera.com> Closes #14153 from srowen/SPARK-16440. (cherry picked from commit 51ade51) Signed-off-by: Sean Owen <sowen@cloudera.com>
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
@@ -416,6 +416,9 @@ class Word2Vec extends Serializable with Logging {
       }
     }
     newSentences.unpersist()
+    expTable.unpersist()
+    bcVocab.unpersist()
+    bcVocabHash.unpersist()
 
     val wordArray = vocab.map(_.word)
     new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)

Original file line number	Diff line number	Diff line change
`@@ -416,6 +416,9 @@ class Word2Vec extends Serializable with Logging {`
`416`	`416`	`}`
`417`	`417`	`}`
`418`	`418`	`newSentences.unpersist()`
	`419`	`+ expTable.unpersist()`
	`420`	`+ bcVocab.unpersist()`
	`421`	`+ bcVocabHash.unpersist()`
`419`	`422`
`420`	`423`	`val wordArray = vocab.map(_.word)`
`421`	`424`	`new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)`