Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calling similar(...) on AbstractCuSparseArray with dims #1184

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
cc65094
calling similar on AbstractCuSparseArray with dims
birkmichael Oct 4, 2021
ad2c4a4
Merge pull request #1 from birkmichael/CuSparse_Similar
birkmichael Oct 4, 2021
2af05e4
robust handling of similar CSC/CSR
birkmichael Oct 4, 2021
25ad0e1
Merge pull request #2 from birkmichael/CuSparse_Similar
birkmichael Oct 4, 2021
53724ce
fixed getrowptr,colvals
birkmichael Oct 4, 2021
8c7a33e
Merge pull request #3 from birkmichael/CuSparse_Similar
birkmichael Oct 4, 2021
6a8b64b
forgot Base. prefix to several instances of similar
birkmichael Oct 4, 2021
ad222bf
Merge pull request #4 from birkmichael/CuSparse_Similar
birkmichael Oct 4, 2021
2620166
used fill instead of CUDA.fill in matrix initialization
birkmichael Oct 4, 2021
14149a2
Merge pull request #5 from birkmichael/CuSparse_Similar
birkmichael Oct 4, 2021
05c85ef
tests for similar, unfinished
birkmichael Oct 5, 2021
309f02e
Created test set for 'similar'
birkmichael Oct 6, 2021
6ec07de
imported some SparseArrays methods
birkmichael Oct 6, 2021
fb4b253
Fixed CSR rowPtr setting in 'similar'
birkmichael Oct 6, 2021
df04acb
tests for 'similar' for CSC,CSR
birkmichael Oct 6, 2021
6956881
Merge pull request #6 from birkmichael/CuSparse_Similar_tests
birkmichael Oct 6, 2021
4bf8e00
splatted 1D tuple dims when creating vector
birkmichael Oct 6, 2021
2d5b613
Merge pull request #7 from birkmichael/vectorfix
birkmichael Oct 6, 2021
faba255
Merge pull request #8 from birkmichael/CuSparse_Similar
birkmichael Oct 8, 2021
1a097db
Added method getindex(CuSparseMat,Colon), changed show for vectors
birkmichael Oct 9, 2021
38cf586
Added conversion from matrix to vector
birkmichael Oct 9, 2021
4b3466e
Merge pull request #9 from birkmichael/CuSparse_similar_toVector
birkmichael Oct 9, 2021
9b1f529
Merge branch 'master' into CuSparse_Similar
birkmichael Oct 9, 2021
575b576
Full indexing of CuSparseArrays
birkmichael Oct 21, 2021
bd16d76
Vectorising matrices
birkmichael Oct 21, 2021
dc4075f
Merge branch 'JuliaGPU:master' into master
birkmichael Oct 22, 2021
69e69d6
Added vectorising of CuSparseMatrices
birkmichael Oct 22, 2021
9a7196f
Update array.jl
birkmichael Oct 22, 2021
f90a4d6
Update conversions.jl
birkmichael Oct 22, 2021
0e45e48
Update conversions.jl
birkmichael Oct 22, 2021
43240b3
Merge pull request #11 from birkmichael/CuSparse_Similar_matrixIndexing
birkmichael Oct 22, 2021
e62cbb6
changed to match updates in master
birkmichael Oct 22, 2021
509d10a
Update conversions.jl
birkmichael Oct 22, 2021
e47b56b
Merge pull request #12 from JuliaGPU/master
birkmichael Oct 22, 2021
66103d1
Merge branch 'CuSparse_Similar' into master
birkmichael Oct 22, 2021
3b64e56
Merge pull request #13 from birkmichael/master
birkmichael Oct 22, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 107 additions & 18 deletions lib/cusparse/array.jl
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixBSR, CuSparseMatrixCO
CuSparseVector

using LinearAlgebra: BlasFloat
using SparseArrays: nonzeroinds, dimlub
using SparseArrays: nonzeroinds, dimlub, getcolptr, rowvals

abstract type AbstractCuSparseArray{Tv, Ti, N} <: AbstractSparseArray{Tv, Ti, N} end
const AbstractCuSparseVector{Tv, Ti} = AbstractCuSparseArray{Tv, Ti, 1}
Expand Down Expand Up @@ -200,6 +200,59 @@ Base.similar(Mat::CuSparseMatrixCSC) = CuSparseMatrixCSC(copy(Mat.colPtr), copy(
Base.similar(Mat::CuSparseMatrixCSR) = CuSparseMatrixCSR(copy(Mat.rowPtr), copy(Mat.colVal), similar(nonzeros(Mat)), Mat.dims)
Base.similar(Mat::CuSparseMatrixBSR) = CuSparseMatrixBSR(copy(Mat.rowPtr), copy(Mat.colVal), similar(nonzeros(Mat)), Mat.blockDim, Mat.dir, nnz(Mat), Mat.dims)

## similar for CSC,CSR with dims, eltype arguments, adapted from SparseArrays.jl. NOTE: calling similar() on a wrapped CuSparseMatrixBSR/COO will result in a CuArray.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe wrap the NOTE on a new line.

But also: why does similar still result in a dense vector? #1184 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't - I spent a lot of time on that. I'll check.

# parent method for similar that preserves stored-entry structure (for when new and old dims match)
function _cusparsesimilar(S::CuSparseMatrixCSC, ::Type{TvNew}, ::Type{TiNew}) where {TvNew,TiNew}
newcolptr = copyto!(similar(getcolptr(S), TiNew), getcolptr(S))
newrowval = copyto!(similar(rowvals(S), TiNew), rowvals(S))
return CuSparseMatrixCSC(newcolptr, newrowval, similar(nonzeros(S), TvNew), size(S))
end
function _cusparsesimilar(S::CuSparseMatrixCSR, ::Type{TvNew}, ::Type{TiNew}) where {TvNew,TiNew}
newrowptr = copyto!(similar(getrowptr(S), TiNew), getrowptr(S))
newcolval = copyto!(similar(colvals(S), TiNew), colvals(S))
return CuSparseMatrixCSR(newrowptr, newcolval, similar(nonzeros(S), TvNew), size(S))
end
# parent methods for similar that preserves only storage space (for when new and old dims differ)
_cusparsesimilar(S::CuSparseMatrixCSC, ::Type{TvNew}, ::Type{TiNew}, dims::Dims{2}) where {TvNew,TiNew} =
CuSparseMatrixCSC(CUDA.fill(one(TiNew), last(dims)+1), similar(rowvals(S), TiNew), similar(nonzeros(S), TvNew),dims)
_cusparsesimilar(S::CuSparseMatrixCSR, ::Type{TvNew}, ::Type{TiNew}, dims::Dims{2}) where {TvNew,TiNew} =
CuSparseMatrixCSR(CUDA.fill(one(TiNew), first(dims)+1), similar(colvals(S), TiNew), similar(nonzeros(S), TvNew),dims)
# parent method for similar that allocates an empty sparse vector (when new dims are single)
_cusparsesimilar(S::CuSparseMatrixCSC, ::Type{TvNew}, ::Type{TiNew}, dims::Dims{1}) where {TvNew,TiNew} =
CuSparseVector(similar(rowvals(S), TiNew, 0), similar(nonzeros(S), TvNew, 0),dims...)
_cusparsesimilar(S::CuSparseMatrixCSR, ::Type{TvNew}, ::Type{TiNew}, dims::Dims{1}) where {TvNew,TiNew} =
CuSparseVector(similar(colvals(S), TiNew, 0), similar(nonzeros(S), TvNew, 0),dims...)

"""
Utility union type of [`CuSparseMatrixCSC`](@ref), [`CuSparseMatrixCSR`](@ref), for which similar is implemented
"""
const CuSparseMatrixCSCR{Tv, Ti} = Union{
CuSparseMatrixCSC{Tv, Ti},
CuSparseMatrixCSR{Tv, Ti}
}
# The following methods hook into the AbstractArray similar hierarchy. The first method
# covers similar(A[, Tv]) calls, which preserve stored-entry structure, and the latter
# methods cover similar(A[, Tv], shape...) calls, which preserve storage space when the shape
# calls for a two-dimensional result.
Base.similar(S::CuSparseMatrixCSCR{<:Any,Ti}, ::Type{TvNew}) where {Ti,TvNew} = _cusparsesimilar(S, TvNew, Ti)
Base.similar(S::CuSparseMatrixCSCR{<:Any,Ti}, ::Type{TvNew}, dims::Union{Dims{1},Dims{2}}) where {Ti,TvNew} =
_cusparsesimilar(S, TvNew, Ti, dims)
# The following methods cover similar(A, Tv, Ti[, shape...]) calls, which specify the
# result's index type in addition to its entry type, and aren't covered by the hooks above.
# The calls without shape again preserve stored-entry structure, whereas those with shape
# preserve storage space when the shape calls for a two-dimensional result.
Base.similar(S::CuSparseMatrixCSCR, ::Type{TvNew}, ::Type{TiNew}) where{TvNew,TiNew} =
_cusparsesimilar(S, TvNew, TiNew)
Base.similar(S::CuSparseMatrixCSCR, ::Type{TvNew}, ::Type{TiNew}, dims::Union{Dims{1},Dims{2}}) where {TvNew,TiNew} =
_cusparsesimilar(S, TvNew, TiNew, dims)
Base.similar(S::CuSparseMatrixCSCR, ::Type{TvNew}, ::Type{TiNew}, m::Integer) where {TvNew,TiNew} =
_cusparsesimilar(S, TvNew, TiNew, (m,))
Base.similar(S::CuSparseMatrixCSCR, ::Type{TvNew}, ::Type{TiNew}, m::Integer, n::Integer) where {TvNew,TiNew} =
_cusparsesimilar(S, TvNew, TiNew, (m, n))

# Handles cases where CSC/CSR are reshaped to more than two dimensions, and all cases above for COO/BSR.
Base.similar(a::AbstractCuSparseArray, ::Type{T}, dims::Base.Dims{N}) where {T,N} =
CuArray{T,N}(undef, dims)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent?

Comment from above also applies.


## array interface

Expand Down Expand Up @@ -238,10 +291,13 @@ Base.eltype(g::CuSparseMatrix{T}) where T = T

SparseArrays.nnz(g::AbstractCuSparseArray) = g.nnz
SparseArrays.nonzeros(g::AbstractCuSparseArray) = g.nzVal

SparseArrays.nonzeroinds(g::AbstractCuSparseVector) = g.iPtr


SparseArrays.getcolptr(g::CuSparseMatrixCSC) = g.colPtr
SparseArrays.rowvals(g::CuSparseMatrixCSC) = g.rowVal
getrowptr(g::CuSparseMatrixCSR) = g.rowPtr
colvals(g::CuSparseMatrixCSR) = g.colVal


LinearAlgebra.issymmetric(M::Union{CuSparseMatrixCSC,CuSparseMatrixCSR}) = false
LinearAlgebra.ishermitian(M::Union{CuSparseMatrixCSC,CuSparseMatrixCSR}) = false
Expand All @@ -260,29 +316,62 @@ Hermitian{T}(Mat::CuSparseMatrix{T}) where T = Hermitian{T,typeof(Mat)}(Mat,'U')

# translations
Base.getindex(A::AbstractCuSparseVector, ::Colon) = copy(A)
Base.getindex(A::AbstractCuSparseMatrix, ::Colon) = CuSparseVector(A)
Base.getindex(A::AbstractCuSparseMatrix, ::Colon, ::Colon) = copy(A)
Base.getindex(A::AbstractCuSparseMatrix, i, ::Colon) = getindex(A, i, 1:size(A, 2))
Base.getindex(A::AbstractCuSparseMatrix, ::Colon, i) = getindex(A, 1:size(A, 1), i)
Base.getindex(A::AbstractCuSparseMatrix, I::Tuple{Integer,Integer}) = getindex(A, I[1], I[2])

# column slices
function Base.getindex(x::CuSparseMatrixCSC, ::Colon, j::Integer)
checkbounds(x, :, j)
r1 = convert(Int, x.colPtr[j])
r2 = convert(Int, x.colPtr[j+1]) - 1
CuSparseVector(rowvals(x)[r1:r2], nonzeros(x)[r1:r2], size(x, 1))
Base.getindex(A::CuSparseMatrixCSC,I::AbstractVector,J::AbstractVector) = CuSparseMatrixCSC(getindex(CuSparseMatrixCSR(getindex(A,:,J)),I,:))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe wrap these two methods for legibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you mean? Write a method for Base.getindex(A::CuSparseMatrixCSCR,I::AbstractVector,J::AbstractVector) that calls either of these two methods, depending on type?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, just line wrapping :-) i.e. adding a newline after the =

Base.getindex(A::CuSparseMatrixCSR,I::AbstractVector,J::AbstractVector) = CuSparseMatrixCSR(getindex(CuSparseMatrixCSC(getindex(A,I,:)),:,J))

function Base.getindex(A::CuSparseMatrixCSR,I::AbstractVector,::Colon)
m,n = size(A)
I_sorted = sort(I);
if !(I_sorted[end] <= m && 1 <= I_sorted[1])
throw(BoundsError())
end

rowptr = getrowptr(A)
vals = nonzeros(A)
cols = colvals(A)

diffVec = vcat(1,diff(rowptr)[I_sorted])
newrowptr = cumsum(diffVec)

indices = map((x,y)->range(x,stop=y),rowptr[I_sorted],rowptr[I_sorted.+1].-1)
@CUDA.allowscalar indices = vcat(indices...) #TODO: figure out how to avoid this step
newval = vals[indices]
newcol = cols[indices]
return CuSparseMatrixCSR(newrowptr,newcol,newval,(length(I_sorted),n))
end

function Base.getindex(x::CuSparseMatrixCSR, i::Integer, ::Colon)
checkbounds(x, i, :)
c1 = convert(Int, x.rowPtr[i])
c2 = convert(Int, x.rowPtr[i+1]) - 1
CuSparseVector(x.colVal[c1:c2], nonzeros(x)[c1:c2], size(x, 2))
function Base.getindex(A::CuSparseMatrixCSC,::Colon,I::AbstractVector)
m,n = size(A)
I_sorted = sort(I);
if !(I_sorted[end] <= n && 1 <= I_sorted[1])
throw(BoundsError())
end

colptr = getcolptr(A)
vals = nonzeros(A)
rows = rowvals(A)

diffVec = vcat(1,diff(colptr)[I_sorted])
newcolptr = cumsum(diffVec)

indices = map((x,y)->range(x,stop=y),colptr[I_sorted],colptr[I_sorted.+1].-1)
@CUDA.allowscalar indices = vcat(indices...) #TODO: figure out how to avoid this step
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CUDA module prefix shouldn't be needed.

But the scalar indexing is not good. where does it come from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm trying to do here is this: get an array indices of vectors, where indices[i] contains a vector of the indices of all elements in nzvals that are in the I_sorted[i] row (or column, depending on CSC or CSR) of the input matrix. After that, indexing becomes simple.

However, the map here returns a CuArray of ranges, and any indexing I do with that will cause a bug, so I need to turn the ranges into vectors first. If I had a method to get a CPU array of CuVectors from the map, instead of a CuArray of ranges, that would be unnecessary. Applying collect.(indices) gives me an "inline variables only" type of error because it tries to instantiate a CuArray of arrays of (sometimes) different sizes. Applying vcat(indices...) gives a scalar indexing error.

The other, more elegant method of solving this, is actually doing what I did in conversions.jl that you commented on - transforming colptr to a CPU array, in which case I can just get an array of arrays using a comprehension or a map.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the map here returns a CuArray of ranges, and any indexing I do with that will cause a bug, so I need to turn the ranges into vectors first.

Could you elaborate? And conversions to CPU data is extremely expensive and to be avoided if possible, so I'd rather look into fixing the bug you're encountering.

newval = vals[indices]
newrow = rows[indices]
CuSparseMatrixCSC(newcolptr,newrow,newval,(m,length(I)))
return CuSparseMatrixCSC(newcolptr,newrow,newval,(m,length(I)))
end

# row slices
Base.getindex(A::CuSparseMatrixCSC, i::Integer, ::Colon) = CuSparseVector(sparse(A[i, 1:end])) # TODO: optimize
Base.getindex(A::CuSparseMatrixCSR, ::Colon, j::Integer) = CuSparseVector(sparse(A[1:end, j])) # TODO: optimize
Base.getindex(x::CuSparseMatrixCSCR, ::Colon, j::Integer) = CuSparseVector(getindex(x,:,[j]))
Base.getindex(x::CuSparseMatrixCSCR, i::Integer ,::Colon) = CuSparseVector(getindex(x,[i],:))
Base.getindex(x::CuSparseMatrixCSCR, I::AbstractVector, j::Integer) = CuSparseVector(getindex(x, I, [j]))
Base.getindex(x::CuSparseMatrixCSCR, i::Integer, J::AbstractVector) = CuSparseVector(getindex(x, [i], J))

function Base.getindex(A::CuSparseMatrixCSC{T}, i0::Integer, i1::Integer) where T
m, n = size(A)
Expand Down Expand Up @@ -446,6 +535,7 @@ Base.copy(Mat::CuSparseMatrixCOO) = copyto!(similar(Mat), Mat)

# input/output


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some needless whitespace changes?

for (gpu, cpu) in [CuSparseVector => SparseVector]
@eval function Base.show(io::IO, ::MIME"text/plain", x::$gpu)
xnnz = length(nonzeros(x))
Expand Down Expand Up @@ -483,7 +573,6 @@ for (gpu, cpu) in [CuSparseMatrixCSC => SparseMatrixCSC,
end
end


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some needless whitespace changes?

# interop with device arrays

function Adapt.adapt_structure(to::CUDA.Adaptor, x::CuSparseVector)
Expand Down
18 changes: 18 additions & 0 deletions lib/cusparse/conversions.jl
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,24 @@ function SparseArrays.sparse(I::CuVector{Cint}, J::CuVector{Cint}, V::CuVector{T
end
end

# CSC to vector
function CuSparseVector(Mat::CuSparseMatrixCSC)
n,m = size(Mat)
dim = n*m
I = 1:m
nzval = nonzeros(Mat)
nzind = rowvals(Mat)
colptr = Array(getcolptr(Mat)) #TODO: figure out how to avoid this step
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems bad. there's no API call for such conversions? or maybe try doing the conversion on the GPU to avoid synchronizing during the copy to/from the CPU?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment Re: scalar indexing above - there I try doing it on the GPU and end up doing scalar indexing. Do you mean API calls to the NVidia CUDA library? I had a look now and couldn't find it, but I will give it a bit more effort later.

indices = map((x,y)->range(x,stop=y),colptr[I],colptr[I.+1].-1)
indices = cu.(collect.(indices))
nzind_array = map(.+,map(inds->nzind[inds],indices) , n.*(0:m-1)) #Vector of cuArrays with indices that should be vectorised into nzind
nzind = vcat(nzind_array...)
return CuSparseVector(nzind,nzval,dim)
end

# CSR, COO, BSR to vector
CuSparseVector(Mat::AbstractCuSparseMatrix) = CuSparseVector(CuSparseMatrixCSC(Mat))


## CSR to CSC

Expand Down
143 changes: 143 additions & 0 deletions test/cusparse/array.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
using CUDA.CUSPARSE
using SparseArrays
using Test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to import Test -- run select tests by doing julia -L test/setup.jl test/cusparse/array.jl

typeSet = [Float32, Float64, ComplexF32, ComplexF64]
n = 10
@testset "similar" begin
@testset "similar(A::$h{$elty},Tv::$newElty)" for elty in typeSet,
newElty in typeSet,
(h,dimPtr,dimVal) in ((CuSparseMatrixCSR,:rowPtr,:colVal), (CuSparseMatrixCSC,:colPtr,:rowVal))

A = sprand(elty, n, n, rand())
dA = h(A)

C_simple = similar(dA)
C_dims = similar(dA,(n,n+1))
C_eltype = similar(dA,newElty)
C_full = similar(dA,newElty,(n,n+1))
@test typeof(C_simple) == typeof(dA)
@test typeof(C_dims) == typeof(dA)
@test (typeof(C_eltype) == typeof(dA) && elty==newElty) || ((typeof(C_eltype) != typeof(dA) && elty!=newElty))
@test (typeof(C_full) == typeof(dA) && elty==newElty) || ((typeof(C_full) != typeof(dA) && elty!=newElty))

properties = Set(propertynames(dA));

conserved_simple = Set([dimPtr,dimVal,:dims,:nnz])
structure_conserved_simple = setdiff(properties,conserved_simple);
@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname simple conserved" for propertyname in conserved_simple
@test getproperty(C_simple,propertyname) == getproperty(dA,propertyname)
end

@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname simple structure conserved" for propertyname in structure_conserved_simple
@test length(getproperty(C_simple,propertyname)) == length(getproperty(dA,propertyname))
@test eltype(getproperty(C_simple,propertyname)) == eltype(getproperty(dA,propertyname))
end

conserved_dims = Set([:nnz])
if h==CuSparseMatrixCSR # Making the array one column longer increases colPtr length but not rowPtr length
structure_conserved_dims = setdiff(properties,union(conserved_dims,Set([dimVal,:dims])))
else #CSC
structure_conserved_dims = setdiff(properties,union(conserved_dims,Set([dimVal,:dims,dimPtr])))
@test length(getproperty(C_dims,dimPtr)) == length(getproperty(dA,dimPtr)) + 1
@test eltype(getproperty(C_dims,dimPtr)) == eltype(getproperty(dA,dimPtr))
end
@test getproperty(C_dims,:dims) == (n,n+1)
@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname dims conserved" for propertyname in conserved_dims
@test getproperty(C_dims,propertyname) == getproperty(dA,propertyname)
end

@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname dims structure conserved" for propertyname in structure_conserved_dims
@test length(getproperty(C_dims,propertyname)) == length(getproperty(dA,propertyname))
@test eltype(getproperty(C_dims,propertyname)) == eltype(getproperty(dA,propertyname))
end

conserved_eltype = Set([:nnz,:dims,dimPtr,dimVal])
structure_conserved_eltype = setdiff(properties,union(conserved_eltype,[:nzVal]))
@test eltype(getproperty(C_eltype,:nzVal)) == newElty
@test length(getproperty(C_eltype,:nzVal)) == length(getproperty(dA,:nzVal))
@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname elty conserved" for propertyname in conserved_eltype
@test getproperty(C_eltype,propertyname) == getproperty(dA,propertyname)
end

@testset "similar(A::$h{$elty},Tv::$newElty) $propertyname elty structure conserved" for propertyname in structure_conserved_eltype
@test length(getproperty(C_eltype,propertyname)) == length(getproperty(dA,propertyname))
@test eltype(getproperty(C_eltype,propertyname)) == eltype(getproperty(dA,propertyname))
end

@testset "similar(A::$h{$elty},Tv::$newElty) full" begin
@test eltype(getproperty(C_full,:nzVal)) == newElty
@test length(getproperty(C_full,:nzVal)) == length(getproperty(dA,:nzVal))
@test h==CuSparseMatrixCSR ? length(getproperty(C_full,dimPtr)) == length(getproperty(dA,dimPtr)) : length(getproperty(C_full,dimPtr)) == length(getproperty(dA,dimPtr))+1
@test getproperty(C_dims,:nnz) == getproperty(dA,:nnz)
@test getproperty(C_full,:dims) == (n,n+1)
end
end

@testset "similar(A::$f($h{$elty}),$newElty)" for elty in typeSet,
newElty in typeSet,
f in (transpose, x->reshape(x,n,n)),
(h,dimPtr,dimVal) in ((CuSparseMatrixCSR,:rowPtr,:colVal), (CuSparseMatrixCSC,:colPtr,:rowVal))

A = sprand(elty, n, n, rand())
dA = f(h(A))

C_simple = similar(dA)
C_dims = similar(dA,(n,n+1))
C_eltype = similar(dA,newElty)
C_full = similar(dA,newElty,(n,n+1))
@test typeof(C_simple) == typeof(parent(dA))
@test typeof(C_dims) == typeof(parent(dA))
@test (typeof(C_eltype) == typeof(parent(dA)) && elty==newElty) || ((typeof(C_eltype) != typeof(parent(dA)) && elty!=newElty))
@test (typeof(C_full) == typeof(parent(dA)) && elty==newElty) || ((typeof(C_full) != typeof(parent(dA)) && elty!=newElty))

properties = Set(propertynames(parent(dA)));

conserved_simple = Set([:nnz])
structure_conserved_simple = setdiff(properties,conserved_simple);
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname simple conserved" for propertyname in conserved_simple
@test getproperty(C_simple,propertyname) == getproperty(parent(dA),propertyname)
end
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname simple structure conserved" for propertyname in structure_conserved_simple
@test length(getproperty(C_simple,propertyname)) == length(getproperty(parent(dA),propertyname))
@test eltype(getproperty(C_simple,propertyname)) == eltype(getproperty(parent(dA),propertyname))
end

conserved_dims = Set([:nnz])
if h==CuSparseMatrixCSR
structure_conserved_dims = setdiff(properties,union(conserved_dims,Set([dimVal,:dims])))
else #CSC
structure_conserved_dims = setdiff(properties,union(conserved_dims,Set([dimVal,:dims,dimPtr])))
@test length(getproperty(C_dims,dimPtr)) == length(getproperty(parent(dA),dimPtr)) + 1
@test eltype(getproperty(C_dims,dimPtr)) == eltype(getproperty(parent(dA),dimPtr))
end
@test getproperty(C_dims,:dims) == (n,n+1)
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname dims conserved" for propertyname in conserved_dims
@test getproperty(C_dims,propertyname) == getproperty(parent(dA),propertyname)
end
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname dims structure conserved" for propertyname in structure_conserved_dims
@test length(getproperty(C_dims,propertyname)) == length(getproperty(parent(dA),propertyname))
@test eltype(getproperty(C_dims,propertyname)) == eltype(getproperty(parent(dA),propertyname))
end

conserved_eltype = Set([:nnz,:dims])
structure_conserved_eltype = setdiff(properties,union(conserved_eltype,[:nzVal]))
@test eltype(getproperty(C_eltype,:nzVal)) == newElty
@test length(getproperty(C_eltype,:nzVal)) == length(getproperty(parent(dA),:nzVal))
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname elty conserved" for propertyname in conserved_eltype
@test getproperty(C_eltype,propertyname) == getproperty(parent(dA),propertyname)
end
@testset "similar(A::$f($h{$elty}),$newElty) $propertyname elty structure conserved" for propertyname in structure_conserved_eltype
@test length(getproperty(C_eltype,propertyname)) == length(getproperty(parent(dA),propertyname))
@test eltype(getproperty(C_eltype,propertyname)) == eltype(getproperty(parent(dA),propertyname))
end
@testset "similar(A::$f($h{$elty}),$newElty) full" begin
@test eltype(getproperty(C_full,:nzVal)) == newElty
@test length(getproperty(C_full,:nzVal)) == length(getproperty(parent(dA),:nzVal))
@test h==CuSparseMatrixCSR ? length(getproperty(C_full,dimPtr)) == length(getproperty(parent(dA),dimPtr)) : length(getproperty(C_full,dimPtr)) == (length(getproperty(parent(dA),dimPtr))+1)
@test getproperty(C_dims,:nnz) == getproperty(parent(dA),:nnz)
@test getproperty(C_full,:dims) == (n,n+1)
end
end


end