-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Widening-based map() to remove return_type #1
base: master
Are you sure you want to change the base?
Conversation
lol
|
Lol on both comments! |
This isn't so good though: julia> map(x->Meta.parse(readline()), v)
1
2.0
ERROR: setindex!() with non-isbitstype eltype is not supported by StaticArrays. Consider using SizedArray. Basically there's a lot less value in the widening machinery transformation unless we can |
Something like |
True though if only it could be called Anyway that brings to mind @tkf's package https://github.com/tkf/BangBang.jl which I'd forgotten about. It's clearly related; maybe it even has all the pieces which are needed for the more general version of this. |
Oh sorry I just assumed you saw andyferris/Freeze.jl#1 Anyway that’s what I was referring to, yes. |
Thanks for the ping :) I believe there is a much better approach to implement map(f, xs) = mapfoldl(f, push!!, xs; init = empty(xs, Union{}))
map(f, xs, args...) = mapfoldl(f, push!!, zip(xs, args..); init = empty(xs, Union{})) (maybe with With this approach, you only need to write a good I agree |
Yes that's cool, thanks! I do like the idea of Or is this missing the point? Are you arguing that
|
I agree. On the implementation level @tkf (ignoring types and just thinking of length information) pre-allocating an For containers where the output |
(Also an aside I'd like to note - the generic code I wrote here not only is as good for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the implementation, is there a way to make widening code more generic, like the idea of using setindex!!
? I suppose you actually want to introduce a function barrier whenever the eltype changes, and continue the loop from there.
Actually... are there other functions (like collect
or copy
) that create a new container and preserve the output keys
but aren't somehow equivalent to map
? copy
is like map(identity)
, right? I'm questioning my assertion above that there is a need to make this pattern generic and reusable (via setindex!!
or something).
@c42f OK, let me try to explain this more clearly. No, this is not what I meant [*1]. I did mean to specialize I agree the semantics of I think we need an abstraction and a set of primitive functions that cleanly separate the properties of data sink and source collections. I think this point is blurred in [*1] OK, it's a very side-note but there is an interesting possibility for optimization to do the inverse of what I call adjoint transformation (convert an iterator transform to transducer; JuliaLang/julia#33526). For example, a partition transducer can be much more efficient when executed as a partition iterator. This is yet another optimization we can do if we build our iteration facility around [*2]
@andyferris Totally agree! I didn't suggest that Note that foldl(pairs(1:3); init=resize!(Union{}[], 3)) do ys, (i, x)
setindex!!(ys, x, i)
end
[*3] OK, this may not be the case if [*4] As you know,
As I just explained, I totally agree that exploiting the property that index set is shared between the data source and sink is important. But, for composable optimal iteration (not just
IIUC the ability for the compiler to do the escape analysis and robustly elide Anyway, we don't have a hard choice here because we can just use
Yes, Just to clarify what I meant by module Demo
foldl(op, xs; init) = _foldl(op, init, xs)
_foldl(op, acc, ::Tuple{}) = acc
_foldl(op, acc, xs::Tuple) = _foldl(op, op(acc, xs[1]), Base.tail(xs))
function _foldl(op, acc, xs::AbstractVector)
isempty(xs) && return acc
acc = op(acc, @inbounds xs[begin])
T = typeof(acc)
for i in firstindex(xs)+1:lastindex(xs)
nxt = op(acc, @inbounds xs[i])
if !(nxt isa T)
return _foldl(op, nxt, @inbounds @view(xs[i+1:end]))
end
acc = nxt
end
return acc
end
push!!(xs::Tuple, x) = (xs..., x)
function push!!(xs::Vector, x)
T = Base.promote_typejoin(eltype(xs), typeof(x))
if T <: eltype(xs)
return push!(xs, x)
else
return push!(copyto!(similar(xs, T), xs), x)
end
end
empty(xs, T) = Base.empty(xs, T)
empty(::Tuple, ::Any) = ()
mapping(f, op) = (acc, x) -> op(acc, f(x))
filtering(f, op) = (acc, x) -> f(x) ? op(acc, x) : acc
map(f, xs) = foldl(mapping(f, push!!), xs; init = empty(xs, Union{}))
filter(f, xs) = foldl(filtering(f, push!!), xs; init = empty(xs, Union{}))
end # module
using Test
using UnPack
@testset begin
@unpack map, filter = Demo
@test map(x -> x + 1, 1:3) == 2:4
@test map(x -> x + 1, (1, 2, 3)) == (2, 3, 4)
@test filter(isodd, 1:3) == [1, 3]
@test filter(isodd, (1, 2, 3)) == (1, 3)
end |
Thanks for the great explanation, especially including the minimal implementation which was really useful to understand what you're getting at. One thing I really like about the minimal implementation you have here is that the recursion in
So to summarize, is the following roughly the default implementation you think should be in map(f, xs) = _map(IteratorSize(xs), f, xs)
# Iterators of unknown length
_map(::SizeUnknown, f, xs) = freeze(foldl(mapping(f, push!!), xs; init = empty(xs, Union{})))
# Iterators with length (also for HasSize?)
function _map(::HasLength, f, xs)
freeze(foldl(pairs(1:3); init=similar(xs, Union{})) do ys, (i, x)
setindex!!(ys, x, i)
end)
end Then in principle we only need to implement The distinction here between |
Yeah, something like that. Though I don't think you need to use intermediate mutable collections in the BTW, I just posted my attempt to make |
@tkf neat :). |
@andyferris Good to know that you find I there is a related discussion with @c42f in Discourse:
|
I came up with this thin-wrapper that I call |
You earn two points https://xkcd.com/356/
It's insane that the compiler removes all the ugly crap I've put in here.