Skip to content

Commit

Permalink
adding example1.cpp, updating readme to reflect this
Browse files Browse the repository at this point in the history
rogersce committed Jun 16, 2011
1 parent 2b888a4 commit 106bfe4
Showing 3 changed files with 87 additions and 48 deletions.
10 changes: 7 additions & 3 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -7,14 +7,18 @@ project(CNPY)

option(ENABLE_STATIC "Build static (.a) library" ON)

ADD_LIBRARY(cnpy SHARED "cnpy.cpp")
add_library(cnpy SHARED "cnpy.cpp")
target_link_libraries(cnpy z)
install(TARGETS "cnpy" LIBRARY DESTINATION lib PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)

if(ENABLE_STATIC)
add_LIBRARY(cnpy-static STATIC "cnpy.cpp")
SET_TARGET_PROPERTIES(cnpy-static PROPERTIES OUTPUT_NAME "cnpy")
add_library(cnpy-static STATIC "cnpy.cpp")
set_target_properties(cnpy-static PROPERTIES OUTPUT_NAME "cnpy")
install(TARGETS "cnpy-static" ARCHIVE DESTINATION lib)
endif(ENABLE_STATIC)

install(FILES "cnpy.h" DESTINATION include)
install(FILES "mat2npz" "npy2mat" DESTINATION bin PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)

add_executable(example1 example1.cpp)
target_link_libraries(example1 cnpy)
57 changes: 12 additions & 45 deletions README
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
Purpose:

Numpy offers the save method for easy saving of arrays into .npy and savez for zipping multiple .npy arrays together into a .npz file. cnpy lets you read and write to these formats in C++. The motivation comes from scientific programming where large amounts of data are generated in C++ and analyzed in Python. Writing to .npy has the advantage of using low-level C++ I/O (fread and fwrite) for speed and binary format for size. The .npy file header takes care of specifying the size, shape, and data type of the array, so specifying the format of the data is unnecessary. Loading data written in numpy formats into C++ is equally simple, but requires you to type-cast the loaded data to the type of your choice.

Installation:

Default installation directory is /usr/local. To specifya different directory, add -DCMAKE_INSTALL_PREFIX=/path/to/install/dir to cmake invocation in step 4.
Default installation directory is /usr/local. To specify a different directory, add -DCMAKE_INSTALL_PREFIX=/path/to/install/dir to the cmake invocation in step 4.

1. get cmake at www.cmake.org
2. create a build directory, say $HOME/build
@@ -9,20 +13,19 @@ Default installation directory is /usr/local. To specifya different directory, a
5. make
6. make install

Using:

To use, #include"cnpy.h" in your source code. Compile the source code mycode.cpp as

g++ -o mycode mycode.cpp -L/path/to/install/dir -lcnpy

Description:

There are two functions for writing data: npy_save, npz_save.

template<typename T> void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w") {
template<typename T> void npz_save(std::string zipname, std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w")

There are 3 functions for reading. npy_load will load a .npy file. npz_load(fname) will load a .npz and return a dictionary of NpyArray structues. npz_load(fname,varname) will load and return the NpyArray for data varname from the specified .npz file.
Note that NpyArray allocates char* data using new[] and *will not* delete the data upon the NpyArray destruction. You are responsible for delete the data yourself.

std::map<std::string,cnpy::NpyArray> cnpy::npz_load(std::string fname)
cnpy::NpyArray cnpy::npz_load(std::string fname, std::string varname)
cnpy::NpyArray cnpy::npy_load(std::string fname)

The data structure for loaded data is below. Data is loaded into a a raw byte array. The array shape and word size are read from the npy header. You are responsible for casting/copying the data to its intended data type.

struct NpyArray {
@@ -31,40 +34,4 @@ struct NpyArray {
unsigned int word_size;
};


Usage examples:

//Write a npy file
int nx = 10, ny = 20;
complex<double>* my_arr = new complex<double>[nx*ny];
const unsigned int shape[] = {ny,nx}; //cnpy assumes "row-major" data, so list the most quickly varying index last!
cnpy::npy_save("my_arr.npy",my_arr,shape,2,"w"); //"w" will overwrite an existing my_arr.npy file

//Append to a npy file
//If the shape of my_arr.npy is currently (n0,n1) and my_append has shape (ny,nx)
//then after appending, my_arry.npy will have shape (n0+ny,nx)
int nx = 10, ny = 20;
float* my_append = new float[nx*ny];
const unsigned int shape[] = {ny,nx};
cnpy::npy_save("my_arr.npy",my_append,shape,2,"a");

//Write an npz file with mixed data
int x = 1;
const unsigned int y_shape[] = {1};
cnpy::npz_save("params.npz","x",&EQUIL,shape,1,"w"); //"w" overwrites any existing file

double y = 2;
const unsigned int y_shape[] = {1};
cnpy::npz_save("params.npz","y",&EQUIL,shape,1,"a"); //"a" appends to existing file

int nx = 10, ny = 20;
float* my_arr = new float[nx*ny];
const unsigned int z_shape[] = {ny,nx};
cnpy::npz_save("params.npz","my_arr",my_arr,shape,2,"a");

//Load an npy file
NpyArray my_arr = npy_load("my_arr.npy");
float* data = reinterpret_cast<float*>(my_arr.data);
assert(my_arr.word_size == sizeof(float))
delete[] data; //you are always responsible for deleting the allocated data

See example1.cpp for examples of how to use the library. example1 will also be build during cmake installation.
68 changes: 68 additions & 0 deletions example1.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#include"cnpy.h"
#include<complex>
#include<cstdlib>
#include<iostream>
#include<map>
#include<string>

const int Nx = 128;
const int Ny = 64;
const int Nz = 32;

int main()
{
//create random data
std::complex<double>* data = new std::complex<double>[Nx*Ny*Nz];
for(int i = 0;i < Nx*Ny*Nz;i++) data[i] = std::complex<double>(rand(),rand());

//save it to file
const unsigned int shape[] = {Nz,Ny,Nx};
cnpy::npy_save("arr1.npy",data,shape,3,"w");

//load it into a new array
cnpy::NpyArray arr = cnpy::npy_load("arr1.npy");
std::complex<double>* loaded_data = reinterpret_cast<std::complex<double>*>(arr.data);

//make sure the loaded data matches the saved data
assert(arr.word_size == sizeof(std::complex<double>));
assert(arr.shape.size() == 3 && arr.shape[0] == Nz && arr.shape[1] == Ny && arr.shape[2] == Nx);
for(int i = 0; i < Nx*Ny*Nz;i++) assert(data[i] == loaded_data[i]);

//append the same data to file
//npy array on file now has shape (Nz+Nz,Ny,Nx)
cnpy::npy_save("arr1.npy",data,shape,3,"a");

//now write to an npz file
//non-array variables are treated as 1D arrays with 1 element
double myVar1 = 1.2;
char myVar2 = 'a';
unsigned int shape2[] = {1};
cnpy::npz_save("out.npz","myVar1",&myVar1,shape2,1,"w"); //"w" overwrites any existing file
cnpy::npz_save("out.npz","myVar2",&myVar2,shape2,1,"a"); //"a" appends to the file we created above
cnpy::npz_save("out.npz","arr1",data,shape,3,"a"); //"a" appends to the file we created above

//load a single var from the npz file
cnpy::NpyArray arr2 = cnpy::npz_load("out.npz","arr1");

//load the entire npz file
typedef std::map<std::string,cnpy::NpyArray> npz_t;
npz_t my_npz = cnpy::npz_load("out.npz");

//check that the loaded myVar1 matches myVar1
cnpy::NpyArray arr_mv1 = my_npz["myVar1"];
double* mv1 = reinterpret_cast<double*>(arr_mv1.data);
assert(arr_mv1.shape.size() == 1 && arr_mv1.shape[0] == 1);
assert(mv1[0] == myVar1);

//cleanup: note that we are responsible for deleting all loaded data

//delete the map of NpyArrays
for(npz_t::iterator it = my_npz.begin(); it != my_npz.end(); ++it)
{
delete[] (*it).second.data;
}

delete[] arr2.data;
delete[] loaded_data;
delete[] data;
}

0 comments on commit 106bfe4

Please sign in to comment.