add support for passing Rust Iterators into JS as iterables

[shaver]

Rust's Iterators underlie a lot of useful patterns, and it would be great to be able to express them to JS for natural interactions as iterables.

This might be the inverse of #1036 ?

[Pauan]

As a workaround, you can use collect() to convert the iterator into a Vec and then send the Vec to JS.

This does cause a minimum of two allocations (one to convert to a Rust Vec and a second to convert to a JS Array).

I'm not sure if that's actually slower than sending an iterator to JS, since iterators would require ping-ponging back and forth many times between Rust and JS (and it has to allocate a JS object once for each element!)

It would be good to get some benchmarks verifying which approach is faster.

[RReverser]

@Pauan Aside from performance difference, it's important to consider a usecase of infinite iterators too - these are not completely uncommon in lazy scenarios, and wouldn't be possible to collect into a temporary Vec before passing to JS.

[RReverser]

Although, rather than solving this particular issue on its own, it seems to be a narrow case of a potentially more general "allow exposing Box<dyn SomeTrait> to JS", which could be done by allowing #[wasm_bindgen] on trait definitions.

[shaver]

Also, working on things piecemeal through an iterator is helpful for avoiding UI jank, whereas computing a few thousand entries in one go to stuff into a Vec<T> can cause UI latency targets to be missed.

It can be seen as an instance of Box<dyn SomeTrait>, but I think it's not just that, because you want them to look like JS iterables, rather than exposing the Rust Iterator API, I think. (Or at least have that option.)

[Pauan]

@RReverser Sure, you're right of course that infinite iterators cannot use Vec.

Also, how would we be able to use #[wasm_bindgen] on trait definitions? The Iterator trait (and impls) are defined in std, so we can't change them.

---

@shaver Ironically, in a large JS library I work on for a company, we used ES6 iterators extensively, but we had to remove all of them, because allocating an object for each element was causing massive UI jank (because of all the garbage collection).

Doing things piecemeal doesn't help, because the main issue isn't raw CPU performance, the main issue is garbage collection (which happens sporadically, and completely locks up the UI for hundreds of milliseconds). That's why game engines obsess so much about memory management.

In my experience it's rare for the CPU to be the bottleneck, garbage collection or the network tend to be much more important.

Of course there are certain situations where the CPU really is the bottleneck, but even then I think a Vec would be faster, because of things like cache locality (and because dealing with a chunk of memory all at once is a lot faster than dealing with many small chunks of memory spread around the heap).

If you have a rather large iterator (say... hundreds of thousands of elements), I think a better solution is to split the iterator into chunks (so you'd make a Vec that contains the first 1,000 elements, then another Vec that contains the next 1,000 elements, etc.)

In fact, if you reused the same Vec, that would mean you would have exactly 1 allocation in Rust, and 1 allocation in JS per chunk. That's a very low number of allocations!

That gives you the best of both worlds: you aren't allocating on every element, so it dramatically minimizes the amount of garbage collection, but it still avoids allocating the entire Vec all at once, so you can handle things in a mostly-piecemeal fashion.

Theoretically this could even handle infinite iterators, though that seems like a more tricky subject.

[nebarf]

Any update on this subject? Are there any workaround to have an iterable on the JS side?

[bovee]

I'm struggling with how to define the iterable interface at all in Rust. I can make a next function, but is there a way to define a function named Symbol.iterator?

I can do RustStruct.prototype[Symbol.iterator] = function() { return this; }; in JS post-build, but it seems like there must be a way to do this in Rust? I'm not even sure this would work if it built, but I've tried #[wasmbindgen(js_name = "@@iterator")] ... which fails to build with "__wasm_bindgen_generated_RustStruct_@@iterator" is not a valid identifier`.

[Pauan]

@bovee You have to use Reflect:

Reflect::set(&object, &Symbol::iterator(), my_function).unwrap();

---

Alternatively you can use inline_js to run whatever JS code you want:

#[wasm_bindgen(inline_js = "
    export function foo(obj) {
        obj[Symbol.iterator] = function () {
            return this;
        };
    }
")]
extern "C" {
    fn foo(obj: &Object);
}

Now you can call foo(&my_object) which will set the Symbol.iterator.

[bovee]

With your second snippet (and finally learning how to run code at init), I was finally able to replace this last bit of JS. Thanks so much!

If anyone's trying to figure out how to manually do this in the future before library support happens, this is what I came up with:

#[wasm_bindgen(start)]
pub fn start() -> Result<(), JsValue> {
    ...
    // This works, but I couldn't figure out how to get the prototype of an object without instantiating a copy first
    foo(&Object::get_prototype_of(&instantiated_object.into()));
    Ok(())
}

[kitty-eu-org]

iterator

Is there a complete code snippet? I don't understand how it is done? Forgive me for being new to js