This project is a library with dataframes implementation. Dataframes are structures allowing more comfortable work with big datasets.
Environment | Build status |
---|---|
CI Build (macOS, Linux, Windows) |
Required dependencies:
- C++ build tools:
- CMake — cross-platform build tool for C++ used by the C++ helper and all its dependencies.
- A mostly C++17-compliant compiler. The tested ones are Visual Studio 2017.8 on Windows and GCC 7.3.0 on Ubuntu. Anything newer is expected to work as well.
- Libraries:
- Apache Arrow — from it C++ library component must be installed
- Boost C++ Libraries — also required by Apache Arrow.
- date library — for calendar support for timestamps.
- pybind11 — C++ Python bindings
- Python 3.6+ with some packages:
matplotlib
seaborn
- RapidJSON — needed for LQuery processing
- {fmt} - version 5.2.0 does not work, use 5.2.1 - C++ library string formatting
Optional dependencies: These dependencies are not required to compile the helper library, however without them certain functionalities shall be disabled.
- xlnt library C++ library — needed for .xlsx file format support. NOTE: On MacOS mwu-tow's fork is needed to fix the compilation issue. On other platforms, official library repo can be used.
- make sure that dependecies are all installed.
- On Mac it is easily done with Anaconda (https://www.anaconda.com/download/).
- Once you have installed it, you can run the following commands to install Arrow:
conda create -n dataframes python=3.6 conda activate dataframes conda install arrow-cpp=0.10.* -c conda-forge conda install pyarrow=0.10.* -c conda-forge conda install rapidjson
- With that in place, you need to instruct CMake where to find the libraries you've just installed. Add the following lines to
native_libs/src/CMakeLists.txt
:set(CMAKE_LIBRARY_PATH "/anaconda3/envs/dataframes/lib") set(CMAKE_INCLUDE_PATH "/anaconda3/envs/dataframes/include")
- build the helper C++ library — CMake will automatically place the built binary in the native_libs/platform directory, so
luna
should out-of-the-box be able to find it.- on Windows start Visual Studio x64 Tools Command Prompt and type:
cd Dataframes\native_libs mkdir build cd build cmake -G"NMake Makefiles" ..\src nmake
- on other platforms:
cd Dataframes/native_libs mkdir build cd build cmake ../src make
Dataframes
refer to the local copy of this repo. - on Windows start Visual Studio x64 Tools Command Prompt and type:
- happily use the dataframes library
The library currently provides wrappers for Apache Arrow structures.
ArrayData
— type-erased storage forArray
consisting of several contiguous memory buffers. The buffer count depends on stored type. Typically there are two buffers: one for values and one for masking nulls. More comples types (union, lists) will use more.Array tag
— data array with strongly typed accessors. See section below for supportedtag
types.ChunkedArray tag
— a list ofArray
s of the same type viewed as a single large array. Allows storing large sequences of data (that could not be feasably stored in a single memory block) and efficient slice / concat operations.Column
— type erased accessor for a namedChunkedArray
. Stored type is represented by using one of its constructors. Described byField
.Table
— ordered sequence ofColumn
s. Described bySchema
.
These types are provided by the library to identify types that can be stored by Array
and their mapping to Luna types. Currently provided type tags are listed in the table below.
Tag type | Luna value type | Apache Arrow type | Memory per element |
---|---|---|---|
StringType | Text | utf8 non-nullable | 4 bytes + 1 byte per character + 1 bit mask |
MaybeStringType | Maybe Text | utf8 nullable | as above |
Int64Type | Int | int64 non-nullable | 8 bytes + 1 bit mask |
MaybeInt64Type | Maybe Int | int64 nullable | as above |
DoubleType | Real | double non-nullable | 8 bytes + 1 bit mask |
MaybeDoubleType | Maybe Real | double nullable | as above |
Note: Arrow's utf8
type is a list of non-nullable bytes.
CSV and Feather files are supported. XLSX files are supported if the helper C++ library was built with XLNT third-part library enabled.
Format | Parser Type | Generator Type | Remarks |
---|---|---|---|
CSV file | CSVParser |
CSVGenerator |
|
XLSX | XLSXParser |
XLSXGenerator |
Requires optional XLNT library |
Feather | FeatherParser |
FeatherGenerator |
Best performance, not all value types are currently supported |
Parser type shall provide the following method:
readFile path :: Text -> Table
Generator type shall provide the following method:writeFile path table :: Text -> Table -> IO None
Column names are by default read from the file. CSV and XLSX parsers can also work with files that do not contain the reader row. In such case one of the methods below should be called:
useCustomNames names
wherenames :: [Text]
are user-provided list of desired column names. If there are more columns in file than names count, then more names will be generated.useGeneratedColumnNames
— all column names will be automatically generated (and the first row will be treated as containing values).
Similarly, the CSV and XLSX generators can be configured whether to output a heading row with names.
setHeaderPolicy WriteHeaderLine
orsetHeaderPolicy SkipHeaderLine
The CSV generator can be also configured whether the fields should be always enclosed within quotes or whether this should be done only when necessary (the latter being the default):
setQuotingPolicy QuoteWhenNeeded
orsetQuotingPolicy QuoteAllFields
DataType
represents the type of values being stored in aArrayData
. Note that this type does not contain information whether it is nullable — being nullable is a property ofField
, notDatatype
.Field
is a namedDataType
with an additional information whether values are nullable. Describes contents of theColumn
.Schema
is a sequence ofField
s describing theTable
.
corr
— calculates correlation matrix, with Pearson correlation coefficient for each column pair.corrWith columnName
— calculates Pearson correlation coefficient between given column in a table and its other columns.countValues columnName
— returns table with pairs (value, count).describeNa
— calculates count of null values and their ratio to the total row count.describe columnName
— calculates a number of statistics for a given column (mean, std, min, quartiles, max).
countMissing
— returns the number of null values in the column.countValues
— counts occurences of each unique value and returns pairs (value, count).- stats:
min
— minimum of values.max
— maximum of values.mean
— mean of values.median
— median of values (interpolated, if value count is even)std
— standard deviation of values.var
— variation of values.sum
— sum of values.quantile q
— value at given quantile, q belongs to <0,1>.
describe
— calculates a number of statistics for a given column (mean, std, min, quartiles, max).
TBD