Stateful Iterator omitting last element when used in list comprehension, crashes on 1-element long vectors.

[racinmat]

Strage behaviour of Stateful iterator:
Although

julia> for a in Iterators.Stateful([1,2,3])
             print(a)
         end
123

works as it should,
With list comprehension it fails to iterate over last element

julia> [a for a in Iterators.Stateful([1,2,3])]
2-element Array{Int64,1}:
 1
 2

julia> [a for a in Iterators.Stateful([1])]
ERROR: BoundsError: attempt to access 0-element Array{Int64,1} at index [1]

tried on Julia 1.3.1, Julia 1.4.0 and Julia 1.4.1.
More details in https://discourse.julialang.org/t/strange-behavior-with-list-comprehensions-using-iterators-stateful/29715

[StefanKarpinski]

cc @Keno since IIRC, you implemented the Stateful iterator

[martinholters]

That looks like a bug in collect for stateful HasLength iterators. It retrieves the first element (to determine its type) by calling iterate once and then invokes _array_for which in the HasLength case calls length to determine the required array size. But the length has at this point decreased by 1 due to the first call to iterate, resulting in an array which is too short by one. I guess this should be restructured to calling length before calling iterate and passing that length on.

[KristofferC]

I came to a similar conclusion in https://discourse.julialang.org/t/strange-behavior-with-list-comprehensions-using-iterators-stateful/29715/2

[martinholters]

Ah yes, should have read that first...

[drdozer]

I think I just hit this:

begin
	items = ["pigs", "dogs", "cats"]
	project(x) = begin println("Projecting: ", x, "!"); join(["P_", x]) end
	projections = Iterators.Stateful(project(x) for x in items)
	println("Testing isempty")
	isempty(projections)
	println("Testing collect")
	collected = collect(projections)
	println("Done")
	
	(items, collected)
end

This produces

"pigs"
"dogs"
"cats",
"P_pigs"
"P_dogs

[drdozer]

I have "fixed" it by adding this:

function Base.collect(si::Iterators.Stateful)
	items = Vector()
 	it = iterate(si)
	while it !== nothing
		i, state = it
		push!(items, i)
		it = iterate(si)
	end
	items
end

It makes my eyes bleed a bit to have to include this in my script though. Although it feels like a plaster over a compound fracture, perhaps it is one approach to fixing this in core?