-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Materialize DimArray
or DimStack
From a Table
#739
Open
JoshuaBillson
wants to merge
40
commits into
rafaqz:main
Choose a base branch
from
JoshuaBillson:materialize
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 3 commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
60256a0
Table Materializer Methods
JoshuaBillson 3526b96
Merged Main
JoshuaBillson eab2fa0
Made col Optional for DimArray
JoshuaBillson d4892df
Apply suggestions from code review
JoshuaBillson ea6751a
Handle coordinates with different loci
JoshuaBillson 13c80da
Merge branch 'materialize' of github.com:JoshuaBillson/DimensionalDat…
JoshuaBillson 6a9d26e
replaced At() with Contains() in _coords_to_ords
JoshuaBillson 9164c22
Added optional selectors and public methods for table materializer
JoshuaBillson 2ebec1c
Updated table constructors for DimArray and DimStack
JoshuaBillson 8e791bf
Updated DimArray and DimStack docs to include table materializer methods
JoshuaBillson 4cd5f9d
Table materializer test cases
JoshuaBillson 0c1991a
export table materializer methods
JoshuaBillson 8758ba9
Merge branch 'rafaqz:main' into materialize
JoshuaBillson 4534de5
Added Random to tables.jl test cases
JoshuaBillson 119fa30
Merge branch 'rafaqz:main' into materialize
JoshuaBillson ed395ca
Update src/array/array.jl
JoshuaBillson 00336af
Update src/table_ops.jl
JoshuaBillson 532f887
Removed exports
JoshuaBillson c98dcb0
Merge branch 'materialize' of github.com:JoshuaBillson/DimensionalDat…
JoshuaBillson 06a2c91
Update src/table_ops.jl
JoshuaBillson 3bacf33
Update src/table_ops.jl
JoshuaBillson 4ced6f7
Update src/table_ops.jl
JoshuaBillson c846dfd
Update src/table_ops.jl
JoshuaBillson fe2c871
Update src/table_ops.jl
JoshuaBillson 61f8220
Replaced selector type with instance.
JoshuaBillson 3d28b43
Merge branch 'materialize' of github.com:JoshuaBillson/DimensionalDat…
JoshuaBillson dbe7b99
Table materializer can now infer dimensions from the coordinates.
JoshuaBillson f410988
Update src/stack/stack.jl
JoshuaBillson a17f069
Update src/table_ops.jl
JoshuaBillson 9bdded9
Update src/table_ops.jl
JoshuaBillson 5451087
Update src/table_ops.jl
JoshuaBillson faf4d76
Update src/table_ops.jl
JoshuaBillson 02f60a3
Update src/table_ops.jl
JoshuaBillson fafd357
Update src/table_ops.jl
JoshuaBillson d7f15f5
Update src/array/array.jl
JoshuaBillson 34a0a69
Update src/table_ops.jl
JoshuaBillson d0b9eb7
Added support for guessing the dimension ordering and span for Dates …
JoshuaBillson 32b0c00
Merge branch 'materialize' of github.com:JoshuaBillson/DimensionalDat…
JoshuaBillson 0ea72a0
Replaced LinRange with StepRangeLen in _build_dim
JoshuaBillson bc62932
Added Tables.istable check to DimArray constructor
JoshuaBillson File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
function _write_vals(data, dims::Tuple, perm, missingval) | ||
# Allocate Destination Array | ||
dst_size = reduce(*, length.(dims)) | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
dst = Vector{eltype(data)}(undef, dst_size) | ||
dst[perm] .= data | ||
|
||
# Handle Missing Rows | ||
_missingval = _cast_missing(data, missingval) | ||
missing_rows = ones(Bool, dst_size) | ||
missing_rows[perm] .= false | ||
return ifelse.(missing_rows, _missingval, dst) | ||
end | ||
|
||
# Find the order of the table's rows according to the coordinate values | ||
_sort_coords(table, dims::Tuple) = _sort_coords(_dim_cols(table, dims), dims) | ||
function _sort_coords(coords::NamedTuple, dims::Tuple) | ||
ords = _coords_to_ords(coords, dims) | ||
indices = _ords_to_indices(ords, dims) | ||
return indices | ||
end | ||
|
||
# Extract coordinate columns from table | ||
function _dim_cols(table, dims::Tuple) | ||
dim_cols = name.(dims) | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return NamedTuple{dim_cols}(Tables.getcolumn(table, col) for col in dim_cols) | ||
end | ||
|
||
# Extract data columns from table | ||
function _data_cols(table, dims::Tuple) | ||
data_cols = _data_col_names(table, dims) | ||
return NamedTuple{Tuple(data_cols)}(Tables.getcolumn(table, col) for col in data_cols) | ||
end | ||
|
||
# Get names of data columns from table | ||
function _data_col_names(table, dims::Tuple) | ||
dim_cols = name.(dims) | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return filter(x -> !(x in dim_cols), Tables.columnnames(table)) | ||
end | ||
|
||
# Determine the ordinality of a set of numerical coordinates | ||
function _coords_to_ords(coords::AbstractVector, dim::AbstractVector{<:Real}) | ||
stride = (last(dim) - first(dim)) / (length(dim) - 1) | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return round.(UInt32, ((coords .- first(dim)) ./ stride) .+ 1) | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
end | ||
|
||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# Determine the ordinality of a set of categorical coordinates | ||
function _coords_to_ords(coords::AbstractVector, dim::AbstractVector) | ||
d = Dict{eltype(dim),UInt32}() | ||
for (i, x) in enumerate(dim) | ||
d[x] = i | ||
end | ||
return map(x -> d[x], coords) | ||
end | ||
|
||
# Preprocessing methods for _coords_to_ords | ||
_coords_to_ords(coords::AbstractVector, dim::Dimension) = _coords_to_ords(coords, collect(dim)) | ||
_coords_to_ords(coords::Tuple, dims::Tuple) = Tuple(_coords_to_ords(c, d) for (c, d) in zip(coords, dims)) | ||
_coords_to_ords(coords::NamedTuple, dims::Tuple) = _coords_to_ords(Tuple(coords[d] for d in name.(dims)), dims) | ||
|
||
# Determine the index from a tuple of coordinate orders | ||
function _ords_to_indices(ords, dims) | ||
stride = 1 | ||
indices = ones(Int, length(ords[1])) | ||
for (ord, dim) in zip(ords, dims) | ||
indices .+= (ord .- 1) .* stride | ||
stride *= length(dim) | ||
end | ||
return indices | ||
end | ||
|
||
function _cast_missing(::AbstractArray{T}, missingval) where {T} | ||
JoshuaBillson marked this conversation as resolved.
Show resolved
Hide resolved
|
||
try | ||
return convert(T, missingval) | ||
catch e | ||
return missingval | ||
end | ||
end |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again we probably need a
Tables.istable
check here