Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-638: [C++] Complex Number Support via ExtensionTypes #10565

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
8e318d0
Complex Number Extension Type
sjperkins Jun 9, 2021
bd2cc0b
lint
sjperkins Jun 9, 2021
2b6f0d1
more linting
sjperkins Jun 9, 2021
1db170d
Copyright headers
sjperkins Jun 9, 2021
9880dc2
Merge branch 'apache:master' into complex-number-extension-support
sjperkins Jun 9, 2021
faa8526
Merge branch 'master' into complex-number-extension-support
sjperkins Jun 14, 2021
5fa7eae
Merge branch 'master' into complex-number-extension-support
sjperkins Jun 15, 2021
e4b9309
[skip ci] WIP
sjperkins Jun 21, 2021
fe1e145
Add Type Traits
sjperkins Jun 21, 2021
76c8ef6
[skip ci] #include complex_type.h
sjperkins Jun 21, 2021
5b72101
Fix header include
sjperkins Jun 21, 2021
0d7524e
Remove Complex{Float,Double} Types in Visitor Actions
sjperkins Jun 21, 2021
ae58a3a
cleanup
sjperkins Jun 21, 2021
dc2125b
Remove COMPLEX_{FLOAT,DOUBLE} from Type
sjperkins Jun 21, 2021
fbe7508
ARROW-13080: [Release] Generate the API docs in ubuntu 20.10
kszucs Jun 15, 2021
5521edf
ARROW-12431: [Python] Mask is inverted when creating FixedSizeBinaryA…
amol- Jun 15, 2021
40585c1
ARROW-13003: [C++] Fix key map unaligned access
cyb70289 Jun 15, 2021
522696b
ARROW-12597: [C++] Enable per-row-group parallelism in async Parquet …
lidavidm Jun 15, 2021
f216d62
ARROW-12096: [C++] Allows users to define arrow timestamp unit for Pa…
isichei Jun 15, 2021
71bcfae
ARROW-13027: [C++] Fix ASAN stack traces in CI
westonpace Jun 15, 2021
322d2c8
ARROW-13085: [Python] Document compatible toolchains for python bindings
kratsg Jun 15, 2021
b33a1a9
ARROW-13082: [CI] Forward R argument to ubuntu-docs build
kszucs Jun 15, 2021
e5d450e
ARROW-13073: [Developer] archery benchmark list: unexpected keyword '…
dianaclarke Jun 16, 2021
4fc96b5
ARROW-11782: [GLib][Ruby][Dataset] Remove bindings for internal classes
kou Jun 16, 2021
cc0bd60
ARROW-13036: [Doc] Mention recommended file extension(s) for Arrow IPC
westonpace Jun 16, 2021
85a4052
ARROW-12995: [C++] Add validation to CSV options
n3world Jun 16, 2021
44495db
ARROW-13090: [Python] Fix create_dir() implementation in FSSpecHandler
pitrou Jun 16, 2021
43bafb8
ARROW-10115: [C++] Add CSV option to treat quoted strings as always n…
pitrou Jun 16, 2021
ca7f9ee
ARROW-12709: [C++] Add binary_join_element_wise
lidavidm Jun 16, 2021
b1dc38c
ARROW-11705: [R] Support scalar value recycling in RecordBatch/Table$…
thisisnic Jun 16, 2021
69d5455
PARQUET-2056: [C++] Add ability for retrieving dictionary and indices…
jp0317 Jun 17, 2021
3cee248
ARROW-12882: [C++][Gandiva] Fix behavior of the convert replace funct…
jpedroantunes Jun 17, 2021
c065178
ARROW-13101: [Python][Doc] pyarrow.FixedSizeListArray does not appear…
westonpace Jun 17, 2021
799a113
ARROW-13092: [C++] Return an error in CreateDir if target is a file
pitrou Jun 17, 2021
1108dce
ARROW-12940: [R] Expose C interface as R6 methods
jonkeane Jun 18, 2021
7a46275
ARROW-12074: [C++][Compute] Add scalar arithmetic kernels for decimal
cyb70289 Jun 18, 2021
877421d
ARROW-13116: [R] Test for RecordBatchReader to C-interface fails on a…
thisisnic Jun 19, 2021
235ed31
ARROW-13042: [C++] Check that kernel output is fully initialized
pitrou Jun 21, 2021
7ecb25e
ARROW-12867: [R] Bindings for abs()
AlenkaF Jun 21, 2021
12f140f
ARROW-13110: [C++] Deadlock can happen when using BackgroundGenerator…
westonpace Jun 21, 2021
453635d
ARROW-13097: [C++] Provide simple reflection utility
bkietz Jun 21, 2021
71cb656
ARROW-13124: [Ruby] Add support for memory view
kou Jun 21, 2021
26b7235
ARROW-13127: [R] Valgrind nightly errors
jonkeane Jun 21, 2021
ed36a1d
cleanup and formatting
sjperkins Jun 22, 2021
4ce447a
fixes
sjperkins Jun 23, 2021
e736161
ARROW-13139: [C++] ReadaheadGenerator cannot be safely copied/moved
westonpace Jun 22, 2021
144cf9e
ARROW-12827: [C++] Improve error message for dataset discovery failure
lidavidm Jun 22, 2021
41be193
ARROW-13034: [Python][Docs] Update the cloud examples on the Parquet …
jorisvandenbossche Jun 22, 2021
0f48742
ARROW-13140: [C++/Python] Upgrade libthrift pin in the nightlies
xhochy Jun 22, 2021
f044d5e
ARROW-12983: [C++][Python][R] Properly overflow to chunked array in P…
kszucs Jun 22, 2021
7164364
ARROW-13135: [C++] Fix Status propagation from Parquet exception
pitrou Jun 22, 2021
3250018
ARROW-10440: [C++][Dataset] Visit FileWriters before Finish
bkietz Jun 22, 2021
a4bd2fa
Merge branch 'master' into complex-number-extension-support
sjperkins Jun 23, 2021
68c1300
remove comment
sjperkins Jun 23, 2021
512fffa
WIP, with segfaults
sjperkins Jun 25, 2021
6a988ad
Merge branch 'master' into complex-number-extension-support
sjperkins Jul 22, 2021
9619e36
Merge branch 'master' into complex-number-extension-support
sjperkins Jul 22, 2021
ada7525
Doesn't segfault, but extension class not registered in Python?
sjperkins Jul 23, 2021
9ffbf56
Merge branch 'master' into complex-number-extension-support
sjperkins Aug 24, 2021
c9abc06
Merge branch 'master' into complex-number-extension-support
sjperkins Nov 2, 2021
7ed676a
[skip ci] Move __arrow_ext_class__ to BaseExtensionType
sjperkins Nov 2, 2021
b3440d5
[skip ci] Progress towards the Pandas API understanding Complex Numbe…
sjperkins Nov 3, 2021
aa580e6
[skip ci] arrow.extension.complex{64,128} -> arrow.complex{64,128}
sjperkins Nov 3, 2021
8f30936
[skip ci] WIP
sjperkins Nov 5, 2021
7ec8d36
[skip ci] Mark converted complex columns
sjperkins Nov 5, 2021
11a285c
[skip ci] Fix up complex -> float conversion
sjperkins Nov 8, 2021
765022e
[skip ci] WIP
sjperkins Nov 8, 2021
e31d556
Fix byte widths for complex types
sjperkins Dec 22, 2021
d741b85
Merge branch 'master' into complex-number-extension-support
sjperkins Dec 22, 2021
7e8dbbb
[skip ci] Add ComplexWriter WIP
sjperkins Dec 22, 2021
f9803a2
Merge branch 'master' into complex-number-extension-support
sjperkins Jan 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ set(ARROW_SRCS
array/diff.cc
array/util.cc
array/validate.cc
extensions/complex_type.cc
builder.cc
buffer.cc
chunked_array.cc
Expand Down
22 changes: 21 additions & 1 deletion cpp/src/arrow/extension_type_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#include "arrow/array/array_nested.h"
#include "arrow/array/util.h"
#include "arrow/extension_type.h"
#include "arrow/extensions/complex_type.h"
#include "arrow/io/memory.h"
#include "arrow/ipc/options.h"
#include "arrow/ipc/reader.h"
Expand Down Expand Up @@ -178,7 +179,9 @@ class ExtStructType : public ExtensionType {

class TestExtensionType : public ::testing::Test {
public:
void SetUp() { ASSERT_OK(RegisterExtensionType(std::make_shared<UuidType>())); }
void SetUp() {
ASSERT_OK(RegisterExtensionType(std::make_shared<UuidType>()));
}

void TearDown() {
if (GetExtensionType("uuid")) {
Expand All @@ -187,6 +190,23 @@ class TestExtensionType : public ::testing::Test {
}
};

TEST_F(TestExtensionType, ComplexTypeTest) {
auto registered_type = GetExtensionType("arrow.complex64");
ASSERT_NE(registered_type, nullptr);

auto type = complex64();
ASSERT_EQ(type->id(), Type::EXTENSION);

const auto& ext_type = static_cast<const ExtensionType&>(*type);
std::string serialized = ext_type.Serialize();

ASSERT_OK_AND_ASSIGN(auto deserialized,
ext_type.Deserialize(fixed_size_list(float32(), 2), serialized));

ASSERT_TRUE(deserialized->Equals(*type));
ASSERT_FALSE(deserialized->Equals(*fixed_size_list(float32(), 2)));
}

TEST_F(TestExtensionType, ExtensionTypeTest) {
auto type_not_exist = GetExtensionType("uuid-unknown");
ASSERT_EQ(type_not_exist, nullptr);
Expand Down
67 changes: 67 additions & 0 deletions cpp/src/arrow/extensions/complex_type.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

// Complex Number Extension Type

#include <mutex>
#include <thread>
#include <sstream>

#include "arrow/extensions/complex_type.h"

namespace arrow {

bool ComplexFloatType::ExtensionEquals(const ExtensionType& other) const {
const auto& other_ext = static_cast<const ExtensionType&>(other);
return other_ext.extension_name() == this->extension_name();
}

bool ComplexDoubleType::ExtensionEquals(const ExtensionType& other) const {
const auto& other_ext = static_cast<const ExtensionType&>(other);
return other_ext.extension_name() == this->extension_name();
}


std::shared_ptr<DataType> complex64() {
return std::make_shared<ComplexFloatType>();
}

std::shared_ptr<DataType> complex128() {
return std::make_shared<ComplexDoubleType>();
}

/// NOTE(sjperkins)
// Suggestions on how to improve this welcome!
std::once_flag complex_float_registered;
std::once_flag complex_double_registered;

Status register_complex_types()
{
std::call_once(complex_float_registered,
RegisterExtensionType,
std::make_shared<ComplexFloatType>());

std::call_once(complex_double_registered,
RegisterExtensionType,
std::make_shared<ComplexDoubleType>());

return Status::OK();
}

static Status complex_types_registered = register_complex_types();

}; // namespace arrow
107 changes: 107 additions & 0 deletions cpp/src/arrow/extensions/complex_type.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

// Complex Number Extension Type
#pragma once

#include <complex>

#include "arrow/extension_type.h"

namespace arrow {


std::shared_ptr<DataType> complex64();
std::shared_ptr<DataType> complex128();


class ComplexFloatArray : public ExtensionArray {
public:
using ExtensionArray::ExtensionArray;
};

class ComplexFloatType : public ExtensionType {
public:
using c_type = std::complex<float>;

explicit ComplexFloatType()
: ExtensionType(fixed_size_list(float32(), 2)) {}

std::string name() const override {
return "complex64";
}

std::string extension_name() const override {
return "arrow.complex64";
}

bool ExtensionEquals(const ExtensionType& other) const override;

std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override {
return std::make_shared<ComplexFloatArray>(data);
}

Result<std::shared_ptr<DataType>> Deserialize(
std::shared_ptr<DataType> storage_type,
const std::string& serialized) const override {
return complex64();
};

std::string Serialize() const override {
return "";
}
};


class ComplexDoubleArray : public ExtensionArray {
public:
using ExtensionArray::ExtensionArray;
};

class ComplexDoubleType : public ExtensionType {
public:
using c_type = std::complex<double>;

explicit ComplexDoubleType()
: ExtensionType(fixed_size_list(float64(), 2)) {}

std::string name() const override {
return "complex128";
}

std::string extension_name() const override {
return "arrow.complex128";
}

bool ExtensionEquals(const ExtensionType& other) const override;

std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override {
return std::make_shared<ComplexFloatArray>(data);
}

Result<std::shared_ptr<DataType>> Deserialize(
std::shared_ptr<DataType> storage_type,
const std::string& serialized) const override {
return complex128();
};

std::string Serialize() const override {
return "";
}
};

}; // namespace arrow
3 changes: 2 additions & 1 deletion cpp/src/arrow/python/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ set(ARROW_PYTHON_SRCS
numpy_to_arrow.cc
python_to_arrow.cc
pyarrow.cc
serialize.cc)
serialize.cc
type_traits.cc)

set_source_files_properties(init.cc PROPERTIES SKIP_PRECOMPILE_HEADERS ON
SKIP_UNITY_BUILD_INCLUSION ON)
Expand Down
Loading