Skip to content

Commit cf5f565

Browse files
committed
Remove special handling for MEM_*_SER
This special handling sacrifices CPU cycles for memory usage by introducing an additional step to deserialize the serialized bytes put into BlockManager. This may cause a performance regression in some cases. For now, let's keep the functionality the same as before, and only include style changes in this PR. This is a precursor to another incoming PR that changes the way unroll RDD partitions.
1 parent 0091ec0 commit cf5f565

File tree

1 file changed

+4
-13
lines changed

1 file changed

+4
-13
lines changed

core/src/main/scala/org/apache/spark/CacheManager.scala

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -143,19 +143,10 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
143143
* to the BlockManager as an iterator and expect to read it back later. This is because
144144
* we may end up dropping a partition from memory store before getting it back, e.g.
145145
* when the entirety of the RDD does not fit in memory. */
146-
if (storageLevel.deserialized) {
147-
val elements = new ArrayBuffer[Any]
148-
elements ++= values
149-
updatedBlocks ++= blockManager.put(key, elements, storageLevel, tellMaster = true)
150-
elements.iterator
151-
} else {
152-
/* This RDD is to be cached in memory in the form of serialized bytes. In this case,
153-
* we only unroll the serialized form of the data, because the deserialized form may
154-
* be much larger and may not fit in memory. */
155-
val bytes = blockManager.dataSerialize(key, values)
156-
updatedBlocks ++= blockManager.putBytes(key, bytes, storageLevel, tellMaster = true)
157-
blockManager.dataDeserialize(key, bytes)
158-
}
146+
val elements = new ArrayBuffer[Any]
147+
elements ++= values
148+
updatedBlocks ++= blockManager.put(key, elements, storageLevel, tellMaster = true)
149+
elements.iterator
159150
}
160151
}
161152
cachedValues.asInstanceOf[Iterator[T]]

0 commit comments

Comments
 (0)