Skip to content

Commit 0d589f4

Browse files
committed
[SPARK-30267][SQL][FOLLOWUP] Use while loop in Avro Array Deserializer
### What changes were proposed in this pull request? This is a follow-up of #26907 It changes the for loop `for (element <- array.asScala)` to while loop ### Why are the changes needed? As per https://github.com/databricks/scala-style-guide#traversal-and-zipwithindex, we should use while loop for the performance-sensitive code. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing tests. Closes #27127 from gengliangwang/SPARK-30267-FollowUp. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>
1 parent 1160457 commit 0d589f4

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -167,13 +167,14 @@ class AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType) {
167167
case (ARRAY, ArrayType(elementType, containsNull)) =>
168168
val elementWriter = newWriter(avroType.getElementType, elementType, path)
169169
(updater, ordinal, value) =>
170-
val array = value.asInstanceOf[java.util.Collection[Any]]
171-
val len = array.size()
172-
val result = createArrayData(elementType, len)
170+
val collection = value.asInstanceOf[java.util.Collection[Any]]
171+
val result = createArrayData(elementType, collection.size())
173172
val elementUpdater = new ArrayDataUpdater(result)
174173

175174
var i = 0
176-
for (element <- array.asScala) {
175+
val iter = collection.iterator()
176+
while (iter.hasNext) {
177+
val element = iter.next()
177178
if (element == null) {
178179
if (!containsNull) {
179180
throw new RuntimeException(s"Array value at path ${path.mkString(".")} is not " +

0 commit comments

Comments
 (0)