Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make partition support non-1-indexed vector. #40830

Merged
merged 15 commits into from
May 21, 2021
Merged

Make partition support non-1-indexed vector. #40830

merged 15 commits into from
May 21, 2021

Conversation

N5N3
Copy link
Member

@N5N3 N5N3 commented May 15, 2021

For a high dimension array, at least OffsetArrays.jl return a 1-indexed vector after reshape, so the current implementation seems ok.

Since the view of Base's Range types has been mapped to getindex correctly, so I think we don't need a separate routine.

For a high dimension array, at least `OffsetArrays.jl` return a 1-indexed vector after reshape, so the current implementation seems ok.

I believe that the view of Base's Range types has been mapped to `getindex` correctly, so I think we don't need a separate routine.
Copy link
Member

@johnnychen94 johnnychen94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are needed to make sure that this PR fixes something. There might be other types, but at least to test the following ones:

  • Base.IdentityUnitRange
  • OffsetArrays.OffsetArray
  • OffsetArrays.IdOffsetRange

@@ -1171,10 +1171,10 @@ function length(itr::PartitionIterator)
return cld(l, itr.n)
end

function iterate(itr::PartitionIterator{<:AbstractRange}, state=1)
state > length(itr.c) && return nothing
function iterate(itr::PartitionIterator{<:AbstractVector}, state = firstindex(itr.c))
Copy link
Member Author

@N5N3 N5N3 May 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear with "invalid", if we add a dispatch for PartitionIterator{<:AbstractRange} , then everything is the same except for replacing view with getindex.

This comment was marked as resolved.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on 1.6.1

julia> UnitRange <: AbstractRange <: AbstractVector
true

something changed in dev?

This comment was marked as resolved.

r = min(state + itr.n - 1, length(itr.c))
return @inbounds itr.c[state:r], r + 1
return @inbounds view(itr.c, state:r), r + 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this change doesn't make a difference, there's no need to change it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the view makes it consistent with the docstring

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit hesitant about this, at least as part of this PR. Since this is supposed to just fix a bug can we remove it for now and perhaps propose that change in a separate PR?

r = min(state + itr.n - 1, length(itr.c))
return @inbounds itr.c[state:r], r + 1
return @inbounds view(itr.c, state:r), r + 1
end

function iterate(itr::PartitionIterator{<:AbstractArray}, state=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the AbstractArray method can be improved, too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, actually, no need. Because here it's using LinearIndexing rules, which always assume 1-based.

@@ -1171,10 +1171,10 @@ function length(itr::PartitionIterator)
return cld(l, itr.n)
end

function iterate(itr::PartitionIterator{<:AbstractRange}, state=1)
state > length(itr.c) && return nothing
function iterate(itr::PartitionIterator{<:AbstractVector}, state = firstindex(itr.c))
Copy link
Member Author

@N5N3 N5N3 May 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear with "invalid", if we add a dispatch for PartitionIterator{<:AbstractRange} , then everything is the same except for replacing view with getindex.

@@ -786,3 +786,8 @@ end
@test b[i] == a[r[i]]
end
end

@testset "proper patition for non-1-indexed vector" begin
@test partition(OffsetArray(1:10,10), 5) |> collect == [1:5,6:10] # OffsetVector
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and also partition(OffsetArray(collect(1:10),10), 5) |> collect. It currently sigfaults with improper index.

function iterate(itr::PartitionIterator{<:AbstractRange}, state=1)
state > length(itr.c) && return nothing
function iterate(itr::PartitionIterator{<:AbstractVector}, state = firstindex(itr.c))
state > lastindex(itr.c) && return nothing
r = min(state + itr.n - 1, length(itr.c))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
r = min(state + itr.n - 1, length(itr.c))
r = min(state + itr.n - 1, lastindex(itr.c))

r = min(state + itr.n - 1, length(itr.c))
return @inbounds itr.c[state:r], r + 1
return @inbounds view(itr.c, state:r), r + 1
end

function iterate(itr::PartitionIterator{<:AbstractArray}, state=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, actually, no need. Because here it's using LinearIndexing rules, which always assume 1-based.

Copy link
Member

@johnnychen94 johnnychen94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current version works for high-dimensional OffsetArray because linear indexing uses 1-based indexing rules. However, I still recommend adding the missing test for it (2d OffsetArray).

test/offsetarray.jl Show resolved Hide resolved
@johnnychen94 johnnychen94 added merge me PR is reviewed. Merge when all tests are passing and removed merge me PR is reviewed. Merge when all tests are passing labels May 15, 2021
N5N3 and others added 3 commits May 16, 2021 08:46
Co-authored-by: Johnny Chen <johnnychen94@hotmail.com>
Add offsetmatrix
@N5N3
Copy link
Member Author

N5N3 commented May 16, 2021

The current version works for high-dimensional OffsetArray because linear indexing uses 1-based indexing rules. However, I still recommend adding the missing test for it (2d OffsetArray).

Well, test for OffsetMatrix has been added.
The failue on Linux64 seems to be brought by a Cholesky factorization of non-positive definite matrix. (So the corresponding testset might be improved?)

@johnnychen94 johnnychen94 added the merge me PR is reviewed. Merge when all tests are passing label May 16, 2021
@simeonschaub simeonschaub removed the merge me PR is reviewed. Merge when all tests are passing label May 19, 2021
@N5N3 N5N3 closed this May 20, 2021
@N5N3 N5N3 reopened this May 20, 2021
@N5N3
Copy link
Member Author

N5N3 commented May 20, 2021

@simeonschaub the dispatch for AbstractRange has been retained.
(I hope I understand your advice correctly)

@simeonschaub
Copy link
Member

Sorry if I missed something above, but why does AbstractVector need a separate method from AbstractArray here? I think it should be fine to remove the special case for Partition{<:AbstractVector} and instead just use firstindex and lastindex in the generic implementation for Partition{<:AbstractArray}.

@N5N3 N5N3 closed this May 20, 2021
N5N3 added 2 commits May 20, 2021 15:15
remove the the special case for abstractvector
@N5N3 N5N3 reopened this May 20, 2021
@N5N3
Copy link
Member Author

N5N3 commented May 20, 2021

That's right, didn't notice that.

Copy link
Member

@simeonschaub simeonschaub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, looks good now! Thanks for the contribution! (BTW, there is no need to close PRs if you are just pushing a new commit)

@simeonschaub simeonschaub merged commit 64940ec into JuliaLang:master May 21, 2021
shirodkara pushed a commit to shirodkara/julia that referenced this pull request Jun 9, 2021
For a high dimension array, at least `OffsetArrays.jl` return a 1-indexed vector after reshape, so the current implementation seems ok.

I believe that the view of Base's Range types has been mapped to `getindex` correctly, so I think we don't need a separate routine.

Co-authored-by: Johnny Chen <johnnychen94@hotmail.com>
johanmon pushed a commit to johanmon/julia that referenced this pull request Jul 5, 2021
For a high dimension array, at least `OffsetArrays.jl` return a 1-indexed vector after reshape, so the current implementation seems ok.

I believe that the view of Base's Range types has been mapped to `getindex` correctly, so I think we don't need a separate routine.

Co-authored-by: Johnny Chen <johnnychen94@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants