Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] JSON tree algorithms refactor II: Constructing device JSON column #16205

Draft
wants to merge 40 commits into
base: branch-24.10
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
1ec9617
added csr data struct
shrshi Jun 11, 2024
022d7ce
formatting
shrshi Jun 11, 2024
382633f
added test
shrshi Jun 25, 2024
1823854
formatting
shrshi Jun 25, 2024
4a7e2a5
Merge branch 'branch-24.08' into json-tree-refactor-ii
shrshi Jun 25, 2024
8d5ddfb
Merge branch 'branch-24.08' into json-tree-refactor
shrshi Jun 26, 2024
84a7749
fixing csr construction
shrshi Jun 28, 2024
810c389
moving the csr algorithms
shrshi Jun 28, 2024
6a1a415
formatting
shrshi Jun 28, 2024
d3468ba
Merge branch 'branch-24.08' into json-tree-refactor-ii
shrshi Jun 28, 2024
85c197d
Merge branch 'branch-24.08' into json-tree-refactor
shrshi Jun 28, 2024
996c6dd
Merge branch 'json-tree-refactor' of github.com:shrshi/cudf into json…
shrshi Jun 28, 2024
3675140
ignoring leaf nodes with non-leaf siblings
shrshi Jul 5, 2024
389df50
formatting
shrshi Jul 6, 2024
4bba629
moving to experimental namespace
shrshi Jul 15, 2024
25530f6
Merge branch 'branch-24.08' into json-tree-refactor
shrshi Jul 15, 2024
df9e65b
formatting
shrshi Jul 15, 2024
d1588c8
removed node properties from csr struct - will be introduced in stage…
shrshi Jul 15, 2024
b809703
partial commit before merge
shrshi Jul 16, 2024
c576370
added two validation checks for column tree
shrshi Jul 17, 2024
b04cebc
formatting
shrshi Jul 17, 2024
b804209
partial work commit
shrshi Jul 21, 2024
b8e8c07
formatting
shrshi Jul 22, 2024
7e1a756
merging branch 24.08 into current branch
shrshi Jul 24, 2024
5541b93
partial commit
shrshi Jul 24, 2024
1490ce9
Merge branch 'branch-24.10' into json-tree-refactor
shrshi Jul 24, 2024
d05e670
better csr construction
shrshi Jul 30, 2024
1ce88be
formatting
shrshi Jul 30, 2024
d6d724c
exec policy is no sync
shrshi Jul 30, 2024
2622d6b
fix copyright year
shrshi Jul 30, 2024
9498372
fixing max row offsets
shrshi Jul 31, 2024
4339b0a
formatting
shrshi Jul 31, 2024
e61288b
Merge branch 'branch-24.10' into json-tree-refactor
shrshi Jul 31, 2024
9b6b7ff
struct docs
shrshi Jul 31, 2024
53db174
Merge branch 'json-tree-refactor' of github.com:shrshi/cudf into json…
shrshi Jul 31, 2024
85608eb
cudf exports!
shrshi Jul 31, 2024
3c21e04
merge after 15979 update
shrshi Aug 1, 2024
3900ee3
refactoring after the csr updates
shrshi Aug 2, 2024
3949cda
minor fixes
shrshi Aug 2, 2024
4d88fe5
formatting
shrshi Aug 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
formatting
  • Loading branch information
shrshi committed Jul 31, 2024
commit 4339b0a1a8a9a5ceee0dc76ceeb11066a8524668
13 changes: 6 additions & 7 deletions cpp/src/io/json/column_tree_construction.cu
Original file line number Diff line number Diff line change
Expand Up @@ -318,8 +318,7 @@ std::tuple<csr, column_tree_properties> reduce_to_column_tree(
auto num_levels = *dev_num_levels_ptr;
list_ancestors[node] = node;
for (int level = 0; level <= num_levels; level++) {
if (list_ancestors[node] > 0)
list_ancestors[node] = colidx[rowidx[list_ancestors[node]]];
if (list_ancestors[node] > 0) list_ancestors[node] = colidx[rowidx[list_ancestors[node]]];
if (list_ancestors[node] == 0 || column_categories[list_ancestors[node]] == NC_LIST)
break;
}
Expand Down Expand Up @@ -455,9 +454,9 @@ reduce_to_column_tree(tree_meta_t& tree,
return !(parent_col_id == parent_node_sentinel ||
column_categories[parent_col_id] == NC_LIST &&
(!is_array_of_arrays || parent_col_id != row_array_parent_col_id));
return (parent_col_id != parent_node_sentinel) &&
(column_categories[parent_col_id] != NC_LIST) ||
(is_array_of_arrays == true && parent_col_id == row_array_parent_col_id);
return (parent_col_id != parent_node_sentinel) &&
(column_categories[parent_col_id] != NC_LIST) ||
(is_array_of_arrays == true && parent_col_id == row_array_parent_col_id);
};

// Mixed types in List children go to different columns,
Expand Down Expand Up @@ -546,5 +545,5 @@ reduce_to_column_tree(tree_meta_t& tree,
std::move(max_row_offsets)};
}

} // namespace detail
} // namespace cudf::io::json
} // namespace detail
} // namespace cudf::io::json
3 changes: 2 additions & 1 deletion cpp/tests/io/json/json_tree_csr.cu
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,8 @@ TEST_F(JsonColumnTreeTests, SimpleLines)
cudf::io::json::experimental::detail::reduce_to_column_tree(
gpu_tree, gpu_col_id, gpu_row_offsets, false, row_array_parent_col_id, stream);

auto iseq = check_equality(d_column_tree, d_max_row_offsets, d_column_tree_csr, d_column_tree_properties, stream);
auto iseq = check_equality(
d_column_tree, d_max_row_offsets, d_column_tree_csr, d_column_tree_properties, stream);
// assert equality between csr and meta formats
assert(iseq == true);
}
Loading