Skip to content

Hdf5 improvements #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 114 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
3b718ee
msvc 2019 and win build script fixes
aothms Mar 5, 2021
60f309d
Create test for HDF libs
johltn Mar 18, 2021
daf138f
Abstract writer class with Hdf writer subclass
johltn Apr 12, 2021
f559f0c
Start iteration through continuous chunk
johltn Apr 12, 2021
6b3baf5
Read bits stream and start with HDF5
johltn Apr 12, 2021
bac0607
Store bit values into HDF file
johltn Apr 26, 2021
dbe4e00
Hyperslab writing on fixed dataset dimensions and add test case
johltn Apr 28, 2021
60d8a6e
Write multidimensional dataset for continuous chunks
johltn Apr 28, 2021
a1ba1c1
Use extents to write multidimensional HDF
johltn May 1, 2021
4e8868a
Handle constant null storage chunks
johltn May 4, 2021
d27df88
Better control flow
johltn May 15, 2021
426cbee
Merge remote-tracking branch 'upstream/master' into HDF5
johltn May 15, 2021
9ff08b9
Count all chunks and make planar explicit
johltn May 16, 2021
e83e84a
Write bits incrementally
johltn Jun 1, 2021
35298b1
Incrementally write dataset regions in Region dataset
johltn Jun 1, 2021
85e9820
New test case
johltn Jun 2, 2021
5433a71
Install hdf5
johltn Jun 2, 2021
e8537f5
Add pathes to include and lib dirs
johltn Jun 2, 2021
290fc67
Add quotes
johltn Jun 2, 2021
162fd65
Add subdir for include path
johltn Jun 2, 2021
8ea1951
Move HDF installation variables
johltn Jun 2, 2021
e7a3c45
Libraries variable
johltn Jun 2, 2021
5db0d4e
Update Travis file
johltn Jun 2, 2021
413febf
Update Travis file
johltn Jun 2, 2021
273444d
serial
johltn Jun 2, 2021
d4c996b
Replace lib name
johltn Jun 2, 2021
39889c0
serial
johltn Jun 2, 2021
55e002a
Update Travis file
johltn Jun 2, 2021
99e967b
Add dl
johltn Jun 3, 2021
e5602bb
Fix iterations over chunks
johltn Jun 6, 2021
952c57e
Fix region slabs dimensions
johltn Jun 6, 2021
c4e64e0
Tentative to incrementally write planar chunks to compound typed dataset
johltn Jun 7, 2021
367f4ba
Planar dataset writing ok for simple example
johltn Jun 7, 2021
e018ef7
Start cleaning before cosntant chunks handling
johltn Jun 7, 2021
7a41f0f
Writing preparation to constant chunks dataset
johltn Jun 7, 2021
95d7c0d
Write constant chunks + region references
johltn Jun 7, 2021
b17dd73
Change dimensions types to comply with clang compilator
johltn Jun 7, 2021
237ddb7
Modify control flow
johltn Jun 8, 2021
1a6c326
Start handling voxels stored as integers
johltn Jun 17, 2021
7bb7411
Store operations as groups containing chunk datasets
johltn Jun 21, 2021
47ed3ad
option to specify padding
aothms Mar 14, 2021
a24873d
Delete appveyor.yml
aothms Mar 14, 2021
c7143b9
Create test for HDF libs
johltn Mar 18, 2021
475fa6c
boolean operation fixes
aothms Mar 21, 2021
d447d40
decribe_components() zero() plane()
aothms Mar 21, 2021
86bfbf4
Add missing include
aothms Mar 21, 2021
7e54760
intersection fixes
aothms Mar 25, 2021
5794a4d
describe_group_by()
aothms Mar 25, 2021
b8eeb33
Use {0,1} voxelset for traversal on PRODUCT_ID method
aothms Mar 26, 2021
562fa5a
Fix hardcoded chunk size
aothms Mar 26, 2021
98e1712
Add chunk size to log output
aothms Mar 26, 2021
9ad66d4
shift() on uint storage
aothms Apr 10, 2021
c0b773d
sweep until
aothms Apr 10, 2021
9d08e2c
constant_like(type=)
aothms Apr 10, 2021
758b9e5
mesh(groups=)
aothms Apr 10, 2021
20fdc74
Remove debugging
aothms Apr 10, 2021
7d0b1cf
don't traverse chunks on max_depth
aothms Apr 11, 2021
8269dd4
uint storage option for traversal
aothms Apr 11, 2021
c6fc6fa
export_csv()
aothms Apr 11, 2021
acba092
Abstract writer class with Hdf writer subclass
johltn Apr 12, 2021
abe3c74
Start iteration through continuous chunk
johltn Apr 12, 2021
a9d47d1
Read bits stream and start with HDF5
johltn Apr 12, 2021
a641a43
initialize num_threads
aothms Apr 25, 2021
d96acbf
don't assign to first when skipped over interval
aothms Apr 25, 2021
e5863c7
voxel storage copy_as() method
aothms Apr 25, 2021
7fc4a33
multi-threaded geometry creation
aothms Apr 25, 2021
8507748
multi threaded VOLUME_PRODUCT_ID
aothms Apr 25, 2021
9e5e7d3
multi-threaded group_by()
aothms Apr 25, 2021
cbfa656
respect with_components in mesh()
aothms Apr 25, 2021
292b870
copy(type=) command
aothms Apr 25, 2021
914aefc
dist: focal
aothms Apr 25, 2021
ba17a9d
Update .travis.yml
aothms Apr 25, 2021
3cda43d
Update .travis.yml
aothms Apr 25, 2021
69a4f1c
Store bit values into HDF file
johltn Apr 26, 2021
1343f77
Hyperslab writing on fixed dataset dimensions and add test case
johltn Apr 28, 2021
7a15675
Write multidimensional dataset for continuous chunks
johltn Apr 28, 2021
c76b91d
don't attempt full chunk union on non-bit types
aothms Apr 29, 2021
18957a4
print_values() debugging option
aothms Apr 29, 2021
e9df48a
more chunk union fixes
aothms Apr 29, 2021
f0cef42
Use extents to write multidimensional HDF
johltn May 1, 2021
12438d1
Handle constant null storage chunks
johltn May 4, 2021
c2dea44
more optimal meshing in case of many chunks
aothms May 11, 2021
cd33f5f
Support for functions and small grammar changes
aothms May 11, 2021
2640dc3
Better control flow
johltn May 15, 2021
8120ab4
Count all chunks and make planar explicit
johltn May 16, 2021
fae86ec
Write bits incrementally
johltn Jun 1, 2021
b34b14a
Incrementally write dataset regions in Region dataset
johltn Jun 1, 2021
d462316
New test case
johltn Jun 2, 2021
7ae6a5f
Install hdf5
johltn Jun 2, 2021
05793b7
Add pathes to include and lib dirs
johltn Jun 2, 2021
e8d4f67
Add quotes
johltn Jun 2, 2021
ac94ad1
Add subdir for include path
johltn Jun 2, 2021
d58742b
Move HDF installation variables
johltn Jun 2, 2021
05dd057
Libraries variable
johltn Jun 2, 2021
d761235
Update Travis file
johltn Jun 2, 2021
94d7c0a
Update Travis file
johltn Jun 2, 2021
80cc06c
serial
johltn Jun 2, 2021
866d716
Replace lib name
johltn Jun 2, 2021
c04b2e4
serial
johltn Jun 2, 2021
1f7a652
Update Travis file
johltn Jun 2, 2021
59281bd
Add dl
johltn Jun 3, 2021
92f1eec
Fix iterations over chunks
johltn Jun 6, 2021
835a060
Fix region slabs dimensions
johltn Jun 6, 2021
d4012d0
Tentative to incrementally write planar chunks to compound typed dataset
johltn Jun 7, 2021
d196c31
Planar dataset writing ok for simple example
johltn Jun 7, 2021
db46840
Start cleaning before cosntant chunks handling
johltn Jun 7, 2021
b387a4c
Writing preparation to constant chunks dataset
johltn Jun 7, 2021
075e55d
Write constant chunks + region references
johltn Jun 7, 2021
6e81854
Change dimensions types to comply with clang compilator
johltn Jun 7, 2021
3d62d8b
Modify control flow
johltn Jun 8, 2021
612ced3
Start handling voxels stored as integers
johltn Jun 17, 2021
b6da6f3
Store operations as groups containing chunk datasets
johltn Jun 21, 2021
43f2dd1
Merge branch 'opensourceBIM:HDF5' into HDF5
johltn Jan 25, 2022
163d991
Voxelization toolkit works with IfcOpenShell v0.7.0 including HDF sup…
johltn Feb 3, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Start cleaning before cosntant chunks handling
  • Loading branch information
johltn authored and aothms committed Jul 10, 2021
commit db468403a755920295931cfc1764cc1347fa19e4
159 changes: 83 additions & 76 deletions writer.h
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,6 @@ class hdf_writer :public abstract_writer {
public:
void Write(const std::string& fnc) {





chunked_voxel_storage<bit_t>* storage = (chunked_voxel_storage<bit_t>*)voxels_;

int continuous_count = 0;
Expand Down Expand Up @@ -127,20 +123,20 @@ class hdf_writer :public abstract_writer {
}
END_LOOP;


const H5std_string FILE_NAME(fnc);
const H5std_string CONTINUOUS_DATASET_NAME("continuous_chunks");
H5::H5File file(FILE_NAME, H5F_ACC_TRUNC);

// Continuous chunks dataset
const H5std_string CONTINUOUS_DATASET_NAME("continuous_chunks");

const int NC = continuous_count;
const int NP = planar_count;

const int NX = storage->chunk_size();
const int NY = storage->chunk_size();
const int NZ = storage->chunk_size();

const int RANK = 4;
H5::H5File file(FILE_NAME, H5F_ACC_TRUNC);


hsize_t dimsf[4];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find all these variable declarations not immensely useful. I know they do it in the samples, but for me it creates an extra indirection between NX and storage->chunk_size(), I'd rather have a more meaningful name for the variables so that we understand the intent.

I'd also prefer the syntax:

hsize_t dimsf[4] = {continuous_count, chunk_size, chunk_size, chunk_size };

And please remove all the unnecessary spaces between type and variable name. If any of them remain after the reorganization.

hsize_t	   NZ
       ^^^^^^

dimsf[RANK] for consistency

With this advise most of the lines can be eliminated making it much more clear for the casual reader what's going on.

dimsf[0] = NC;
dimsf[1] = NX;
Expand All @@ -152,7 +148,7 @@ class hdf_writer :public abstract_writer {
datatype.setOrder(H5T_ORDER_LE);
H5::DataSet dataset = file.createDataSet(CONTINUOUS_DATASET_NAME, datatype, dataspace);

// Hyperslab prep
// Continuous dataset hyperslab preparation
hsize_t offset[4];
offset[0] = -1;
offset[1] = 0;
Expand All @@ -162,103 +158,87 @@ class hdf_writer :public abstract_writer {
hsize_t slab_dimsf[4] = { 1, 1, 1, 1 };
H5::DataSpace mspace2(RANK, slab_dimsf);

// Regions dataset
const H5std_string REGIONS_DATASET_NAME("regions");
const int REGION_RANK = 1;
hsize_t regions_dimsf[1];
regions_dimsf[0] = continuous_count + constant_count + planar_count;
//regions_dimsf[1] = 1;
H5::DataSpace regions_dataspace(REGION_RANK, regions_dimsf);
H5::PredType regions_datatype(H5::PredType::STD_REF_DSETREG);
regions_datatype.setOrder(H5T_ORDER_LE);

H5::DataSet regions_dataset = file.createDataSet(REGIONS_DATASET_NAME, regions_datatype, regions_dataspace);
// Regions hyperlab
hsize_t region_offset[1];
region_offset[0] = -1;
//region_offset[1] = 0;
hsize_t region_slab_dimsf[1] = { 1 };
H5::DataSpace mspace3(REGION_RANK, region_slab_dimsf);

// Chunk hyperslab
hsize_t chunk_offset[4];
const int CHUNK_RANK = 4;
chunk_offset[0] = -1;
chunk_offset[1] = 0;
chunk_offset[2] = 0;
chunk_offset[3] = 0;
hsize_t chunk_dimsf[4] = { 1, NX, NY, NZ };
H5::DataSpace chunk_space(CHUNK_RANK, chunk_dimsf);


//Planar chunks dataset
typedef struct s2_t {
int a;
hvl_t b;
int axis;
hvl_t offsets;
} s2_t;

const H5std_string AXIS("axis");
const H5std_string OFFSETS("offsets");

H5::VarLenType varlen_type(&H5::PredType::NATIVE_INT);
H5::CompType mtype2(sizeof(s2_t));
mtype2.insertMember(AXIS, HOFFSET(s2_t, a), H5::PredType::NATIVE_INT);
mtype2.insertMember(OFFSETS, HOFFSET(s2_t, b), varlen_type);
mtype2.insertMember(AXIS, HOFFSET(s2_t, axis), H5::PredType::NATIVE_INT);
mtype2.insertMember(OFFSETS, HOFFSET(s2_t, offsets), varlen_type);
const H5std_string PLANAR_DATASET_NAME("planar_chunks");


const int PLANAR_RANK = 1;
hsize_t planar_dimsf[1];
planar_dimsf[0] = planar_count;
//planar_dimsf[1] = 1;


H5::DataSpace planar_dataspace(PLANAR_RANK, planar_dimsf);

H5::DataSet planar_dataset = file.createDataSet(PLANAR_DATASET_NAME, mtype2, planar_dataspace);



hsize_t planar_offset[1];
planar_offset[0] = 1;


hsize_t planar_dims[2] = { 1,1};
planar_offset[0] = -1;
hsize_t planar_dims[1] = { 1 };
H5::DataSpace planar_space(PLANAR_RANK, planar_dims);

planar_dataspace.selectHyperslab(H5S_SELECT_SET, planar_dims, planar_offset);


// Regions dataset
const H5std_string REGIONS_DATASET_NAME("regions");

std::vector<s2_t> input_vec;
hsize_t regions_dimsf[1];
regions_dimsf[0] = continuous_count + constant_count + planar_count;

hvl_t varlen_offsets;
std::vector<int> tt = { 1,2,4,4,4,4 };
varlen_offsets.p = tt.data();
const int REGION_RANK = 1;

H5::DataSpace regions_dataspace(REGION_RANK, regions_dimsf);
H5::PredType regions_datatype(H5::PredType::STD_REF_DSETREG);
regions_datatype.setOrder(H5T_ORDER_LE);
H5::DataSet regions_dataset = file.createDataSet(REGIONS_DATASET_NAME, regions_datatype, regions_dataspace);

// Regions dataset hyperslab preparation
hsize_t region_offset[1];
region_offset[0] = -1;
hsize_t region_slab_dimsf[1] = { 1 };
H5::DataSpace mspace3(REGION_RANK, region_slab_dimsf);

// No bug but nothing written when inspecting output file
s2_t ss;
ss.a = 4;
ss.b = varlen_offsets;
input_vec.push_back(ss);
planar_dataset.write(&ss, mtype2, planar_space, planar_dataspace);
// Continuous chunks region hyperslab preparation
hsize_t chunk_offset[4];
const int CHUNK_RANK = 4;
chunk_offset[0] = -1;
chunk_offset[1] = 0;
chunk_offset[2] = 0;
chunk_offset[3] = 0;
hsize_t chunk_dimsf[4] = { 1, NX, NY, NZ };
H5::DataSpace chunk_space(CHUNK_RANK, chunk_dimsf);

// Planar chunks region hyperslab preparation
hsize_t planar_chunk_offset[1];
const int PLANAR_CHUNK_RANK = 1;
planar_chunk_offset[0] = 1;
hsize_t planar_chunk_dimsf[1] = { 1 };
H5::DataSpace planar_chunk_space(PLANAR_CHUNK_RANK, planar_chunk_dimsf);


BEGIN_LOOP(size_t(0), nchunks_x, 0U, nchunks_y, 0U, nchunks_z)
auto c = storage->get_chunk(ijk);

if (c == nullptr) {
std::cout << "Null pointer" << std::endl;
if (c == nullptr) {
std::cout << "Null pointer" << std::endl;

}
}

else {
if (c->is_explicit() || c->is_constant()) {
if (c->is_constant()) {
else {
if (c->is_explicit() || c->is_constant()) {
if (c->is_constant()) {
std::cout << "Constant chunk" << std::endl;

}

else {
std::cout << "Continuous chunk" << std::endl;

Expand Down Expand Up @@ -294,18 +274,45 @@ class hdf_writer :public abstract_writer {
}
}

else {
std::cout << "Planar chunk" << std::endl;
planar_voxel_storage<bit_t>* planvox = (planar_voxel_storage<bit_t>*)c;
else {
std::cout << "Planar chunk" << std::endl;
planar_voxel_storage<bit_t>* planvox = (planar_voxel_storage<bit_t>*)c;

auto axis = planvox->axis();
auto off = planvox->offsets();

std::vector<int> offsets(off.begin(), off.end());

auto off = planvox->offsets();
auto axis = planvox->axis();
planar_offset[0]++;

planar_dataspace.selectHyperslab(H5S_SELECT_SET, planar_dims, planar_offset);
std::vector<s2_t> input_vec;
hvl_t varlen_offsets;


varlen_offsets.len = offsets.size();

varlen_offsets.p = offsets.data();
s2_t compound_data;
compound_data.axis = axis;
compound_data.offsets = varlen_offsets;
input_vec.push_back(compound_data);
planar_dataset.write(&compound_data, mtype2, planar_space, planar_dataspace);


region_offset[0]++;
regions_dataspace.selectHyperslab(H5S_SELECT_SET, region_slab_dimsf, region_offset);

planar_chunk_offset[0]++;

dataspace.selectHyperslab(H5S_SELECT_SET, planar_chunk_dimsf, planar_chunk_offset);

hobj_ref_t inter[1];
file.reference(&inter[0], "/planar_chunks", planar_dataspace, H5R_DATASET_REGION);
regions_dataset.write(inter, H5::PredType::STD_REF_DSETREG, mspace3, regions_dataspace);

}
}
}
END_LOOP
}
};
Expand Down