Skip to content

Fix Stream.iterator memory leak #9710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 27, 2021

Conversation

htmldoug
Copy link
Contributor

@htmldoug htmldoug commented Jul 26, 2021

Stream.iterator kept a reference to the head of the stream which prevented any part of the Stream from getting garbage collected. I suspect this was the underlying cause of similar fixes like #8367. Scala 2.11 is unaffected.

This change prevents LinearSeqIterator from creating a private field and closing over the head of the Stream as a coll class field.

Before:

javap -p build/quick/classes/library/scala/collection/LinearSeqIterator.class
Compiled from "LinearSeq.scala"
public final class scala.collection.LinearSeqIterator<A> extends scala.collection.AbstractIterator<A> {
  private final scala.collection.LinearSeqOps<A, scala.collection.LinearSeq, scala.collection.LinearSeq<A>> coll;
  private scala.collection.LinearSeqIterator<A>.LazyCell these;
  public boolean hasNext();
  public A next();
...
}

After:

javap -p build/quick/classes/library/scala/collection/LinearSeqIterator.class
Compiled from "LinearSeq.scala"
public final class scala.collection.LinearSeqIterator<A> extends scala.collection.AbstractIterator<A> {
  private scala.collection.LinearSeqIterator<A>.LazyCell these;
  public boolean hasNext();
  public A next();
...
}

I didn't find any open issues on scala/bug for this.

@scala-jenkins scala-jenkins added this to the 2.13.7 milestone Jul 26, 2021
@SethTisue SethTisue added the library:collections PRs involving changes to the standard collection library label Jul 27, 2021
@SethTisue
Copy link
Member

review by @scala/collections ?

@@ -8,4 +8,5 @@ object Test extends App {
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).iterator.foreach(_ => ())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did this test fail before the change?

Just making sure we have a test to catch any regression.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, fails without the change. OOME.

Copy link
Member

@lrytz lrytz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 thank you!

@lrytz lrytz merged commit fd3fb31 into scala:2.13.x Jul 27, 2021
@htmldoug htmldoug deleted the 2.13.x-Stream-iterator-leak branch July 27, 2021 08:13
@@ -8,4 +8,5 @@ object Test extends App {
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Stream.tabulate(100)(_ => new Array[AnyRef](10000)).collectFirst { case x if false => x }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

late to the party, but why is this line repeated 3 times?

Copy link
Contributor Author

@htmldoug htmldoug Jul 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wondered, too. git says two of them were added in https://github.com/scala/scala/pull/9324/files#diff-b0ab17467318c0ab594f748c21870010636257c36577157fcdd7f01836f4b272. The same PR also updated the number of deprecation warnings, so it seems somewhat intentional at least.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's funny - I have no recollection doing that. Was I trying to stress the GC? Did I do that before changing the java options and then committing? I'd assume some sanity and say it's related to using JDK 15 in some way... Maybe someone wants to try to retrace my steps. I also don't mind if someone wants to remove the extras.

hamzaremmal pushed a commit to hamzaremmal/scala3 that referenced this pull request May 2, 2025
…tor-leak

Fix Stream.iterator memory leak
hamzaremmal pushed a commit to scala/scala3 that referenced this pull request May 7, 2025
…tor-leak

Fix Stream.iterator memory leak
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
library:collections PRs involving changes to the standard collection library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants