Skip to content

GH-23221: [C++] Add support for building with Emscripten #37821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 110 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
5503984
cmake changes for emscripten
joemarshall Sep 21, 2023
331aaa3
reverted cmakelists from python
joemarshall Sep 21, 2023
3e4156e
documentation tidying after review
joemarshall Sep 25, 2023
80b21d1
preset fixes
joemarshall Sep 25, 2023
908abf9
updates to simplify emscripten build process
joemarshall Sep 25, 2023
6effae5
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Sep 25, 2023
31ca7f9
build fixes
joemarshall Sep 25, 2023
c26d68c
better comments
joemarshall Sep 25, 2023
f5ec444
cmake fixes for python emscripten build to work
joemarshall Sep 27, 2023
3c2f179
fix pyodide stdc version number
joemarshall Sep 27, 2023
aec8a79
made substrait, ipc, brotli build on emscripten
joemarshall Sep 28, 2023
0db01d7
docs fix
joemarshall Sep 28, 2023
0daa670
bad if statement fix
joemarshall Sep 28, 2023
ff58481
cmake-format
joemarshall Sep 28, 2023
008b3dc
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Oct 2, 2023
20f9f05
removed python cmake changes from emscripten C++ build pr
joemarshall Oct 2, 2023
b777f69
documentation fix
joemarshall Oct 2, 2023
d93d8cc
preset fixes
joemarshall Oct 2, 2023
831715d
toolchain fixes
joemarshall Oct 2, 2023
3fd8a1c
typo
joemarshall Oct 2, 2023
28f675d
fix for webassembly where size_t != int64_t
joemarshall Oct 2, 2023
97e58f9
size_t not always == in64_t
joemarshall Oct 2, 2023
a00f068
reverting size_t fixes
joemarshall Oct 2, 2023
bda2b26
debug build fixes
joemarshall Oct 2, 2023
14b4968
fixed order of build preset dependencies
joemarshall Oct 2, 2023
834b5a3
debug build fixes
joemarshall Oct 4, 2023
ff2a7c9
cmake format
joemarshall Oct 4, 2023
45ece1e
typos etc. from code review
joemarshall Oct 4, 2023
28ccba9
review fixes
joemarshall Oct 4, 2023
7a8d71a
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Oct 4, 2023
adc68e8
docs fixes
joemarshall Oct 4, 2023
7cb0da9
comments grammar
joemarshall Oct 4, 2023
e3f1a81
lint
joemarshall Oct 4, 2023
123813f
correct separators in externalproject call
joemarshall Oct 9, 2023
9fff2ce
test changes for emscripten
joemarshall Oct 9, 2023
0b5cc3b
emscripten archery fixes
joemarshall Oct 11, 2023
b5fedaa
force non-parallel emscripten debug build
joemarshall Oct 11, 2023
76d464b
split dwarf debug info on emscripten
joemarshall Oct 12, 2023
b2dd85b
use limited debug info on emscripten builds
joemarshall Oct 13, 2023
8d3b255
disable test that won't work on emscripten
joemarshall Oct 13, 2023
8b7d2e4
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Oct 13, 2023
84867d2
set debug flag to g2
joemarshall Oct 13, 2023
9c7ad2e
lint
joemarshall Oct 13, 2023
275c08a
typo
joemarshall Oct 13, 2023
22ace85
test fixes
joemarshall Oct 16, 2023
527e477
fixes to tests for emscripten
joemarshall Oct 18, 2023
a96f793
emscripten test fixes
joemarshall Oct 18, 2023
5e8ee70
more test fixes
joemarshall Oct 18, 2023
ae1c579
fix to tests for emscripten
joemarshall Oct 18, 2023
688d1ab
lint
joemarshall Oct 19, 2023
da548e6
formatting
joemarshall Oct 19, 2023
b457d7c
dockerfile fixes in code review
joemarshall Oct 19, 2023
92ead7c
review updates
joemarshall Oct 19, 2023
2a1d67f
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Oct 19, 2023
58fbdfe
cmake-format updated version
joemarshall Oct 19, 2023
e9e1988
lint
joemarshall Oct 19, 2023
ddbcf56
licence on test init js
joemarshall Oct 19, 2023
1d4cc2c
fix to preprocessor directive check
joemarshall Oct 19, 2023
e3631fd
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Oct 19, 2023
8b8f8d0
fix #defines
joemarshall Oct 27, 2023
63d497a
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Oct 27, 2023
6124d53
review fixes
joemarshall Nov 7, 2023
b10b415
use __EMSCRIPTEN__ because emscripten docs say so
joemarshall Nov 7, 2023
09bffa0
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Nov 7, 2023
c9a79cc
whitespace
joemarshall Nov 7, 2023
4980d48
build fixes after review
joemarshall Nov 7, 2023
d6026f9
fix zlib on wasm
joemarshall Nov 7, 2023
fed5439
fix to csv test
joemarshall Nov 8, 2023
48c795b
typo #if! instead of ifndef
joemarshall Nov 8, 2023
4f2ebfb
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Nov 8, 2023
7b01717
revert test changes
joemarshall Nov 8, 2023
f780784
include config.h for threading define
joemarshall Nov 8, 2023
d23f1a1
lint
joemarshall Nov 8, 2023
bdd051d
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Nov 8, 2023
f79e49f
dead code elimination in test executables
joemarshall Nov 8, 2023
232d7f5
revert main_module change
joemarshall Nov 8, 2023
5b65a36
default to -O1 for emscripten
joemarshall Nov 9, 2023
72481cf
lint
joemarshall Nov 9, 2023
57e1b65
Apply suggestions from code review
joemarshall Nov 22, 2023
9506ca5
use -sRELOCATABLE instead of -fPIC
joemarshall Nov 22, 2023
617ba51
remove emscriptenoverrides
joemarshall Nov 22, 2023
dd2c662
review changes
joemarshall Nov 22, 2023
b1050f5
Update cpp/src/arrow/util/io_util.cc
joemarshall Nov 22, 2023
145426f
restrict number of concurrent processes in emscripten
joemarshall Nov 22, 2023
064659c
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Nov 22, 2023
8cc5ae3
lint fixes
joemarshall Nov 24, 2023
2429261
Apply suggestions from code review
joemarshall Mar 21, 2024
370b1ff
Apply suggestions from code review
joemarshall Mar 21, 2024
d0fa881
Apply suggestions from code review
joemarshall Mar 21, 2024
da176ad
documentation consistency
joemarshall Mar 21, 2024
5f058c9
don't ignore conversion errors any more
joemarshall Mar 21, 2024
51705de
build fixes from review
joemarshall Mar 21, 2024
225cef4
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Mar 21, 2024
3179e57
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Mar 21, 2024
d3e0770
typo
joemarshall Mar 21, 2024
abcd7df
fix script permissions
joemarshall Mar 21, 2024
cc0c1d5
toolchain lint
joemarshall Mar 22, 2024
701d5af
test commit:
joemarshall Mar 25, 2024
0064fa0
reverted test commit
joemarshall Mar 25, 2024
91a6e36
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall Mar 27, 2024
07f2268
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall Mar 27, 2024
028682e
Apply suggestions from code review
joemarshall Mar 28, 2024
d36fca2
code review changes
joemarshall Mar 28, 2024
ee99e5e
toolchain build fix
joemarshall Mar 28, 2024
b73740a
cmake fixes for emscripten
joemarshall Mar 28, 2024
4dc3d03
alphabetical order oops
joemarshall Mar 29, 2024
8b07ebf
remove dynamic linking test on emsripten
joemarshall Mar 29, 2024
c0692da
lint
joemarshall Mar 29, 2024
d3bc3b4
Use arrow/util/config.h
kou Apr 4, 2024
1307910
Remove trailing spaces
kou Apr 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion ci/docker/ubuntu-22.04-cpp.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ RUN latest_system_llvm=14 && \
RUN apt-get update -y -q && \
apt-get install -y -q --no-install-recommends \
autoconf \
bzip2 \
ca-certificates \
ccache \
cmake \
Expand Down Expand Up @@ -115,10 +116,20 @@ RUN apt-get update -y -q && \
rapidjson-dev \
rsync \
tzdata \
wget && \
wget \
xz-utils && \
apt-get clean && \
rm -rf /var/lib/apt/lists*

# install emscripten using EMSDK
ARG emscripten_version="3.1.45"
RUN cd ~ && git clone https://github.com/emscripten-core/emsdk.git && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, two things:

  1. it would have been better for resource usage to make a separate dockerfile for this, perhaps
  2. could this be a script to avoid having tons of hand-coded bash code in dockerfiles?

cd emsdk && \
./emsdk install ${emscripten_version} && \
./emsdk activate ${emscripten_version} && \
echo "Installed emsdk to:" ~/emsdk


ARG gcc_version=""
RUN if [ "${gcc_version}" = "" ]; then \
apt-get update -y -q && \
Expand Down Expand Up @@ -151,6 +162,9 @@ RUN if [ "${gcc_version}" = "" ]; then \
update-alternatives --set c++ /usr/bin/g++; \
fi

# make sure zlib is cached in the EMSDK folder
RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why this isn't together with the install step above?


COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local

Expand Down
232 changes: 126 additions & 106 deletions ci/scripts/cpp_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -92,112 +92,132 @@ esac
mkdir -p ${build_dir}
pushd ${build_dir}

cmake \
-Dabsl_SOURCE=${absl_SOURCE:-} \
-DARROW_ACERO=${ARROW_ACERO:-OFF} \
-DARROW_AZURE=${ARROW_AZURE:-OFF} \
-DARROW_BOOST_USE_SHARED=${ARROW_BOOST_USE_SHARED:-ON} \
-DARROW_BUILD_BENCHMARKS_REFERENCE=${ARROW_BUILD_BENCHMARKS:-OFF} \
-DARROW_BUILD_BENCHMARKS=${ARROW_BUILD_BENCHMARKS:-OFF} \
-DARROW_BUILD_EXAMPLES=${ARROW_BUILD_EXAMPLES:-OFF} \
-DARROW_BUILD_INTEGRATION=${ARROW_BUILD_INTEGRATION:-OFF} \
-DARROW_BUILD_SHARED=${ARROW_BUILD_SHARED:-ON} \
-DARROW_BUILD_STATIC=${ARROW_BUILD_STATIC:-ON} \
-DARROW_BUILD_TESTS=${ARROW_BUILD_TESTS:-OFF} \
-DARROW_BUILD_UTILITIES=${ARROW_BUILD_UTILITIES:-ON} \
-DARROW_COMPUTE=${ARROW_COMPUTE:-ON} \
-DARROW_CSV=${ARROW_CSV:-ON} \
-DARROW_CUDA=${ARROW_CUDA:-OFF} \
-DARROW_CXXFLAGS=${ARROW_CXXFLAGS:-} \
-DARROW_CXX_FLAGS_DEBUG="${ARROW_CXX_FLAGS_DEBUG:-}" \
-DARROW_CXX_FLAGS_RELEASE="${ARROW_CXX_FLAGS_RELEASE:-}" \
-DARROW_CXX_FLAGS_RELWITHDEBINFO="${ARROW_CXX_FLAGS_RELWITHDEBINFO:-}" \
-DARROW_C_FLAGS_DEBUG="${ARROW_C_FLAGS_DEBUG:-}" \
-DARROW_C_FLAGS_RELEASE="${ARROW_C_FLAGS_RELEASE:-}" \
-DARROW_C_FLAGS_RELWITHDEBINFO="${ARROW_C_FLAGS_RELWITHDEBINFO:-}" \
-DARROW_DATASET=${ARROW_DATASET:-OFF} \
-DARROW_DEPENDENCY_SOURCE=${ARROW_DEPENDENCY_SOURCE:-AUTO} \
-DARROW_ENABLE_THREADING=${ARROW_ENABLE_THREADING:-ON} \
-DARROW_ENABLE_TIMING_TESTS=${ARROW_ENABLE_TIMING_TESTS:-ON} \
-DARROW_EXTRA_ERROR_CONTEXT=${ARROW_EXTRA_ERROR_CONTEXT:-OFF} \
-DARROW_FILESYSTEM=${ARROW_FILESYSTEM:-ON} \
-DARROW_FLIGHT=${ARROW_FLIGHT:-OFF} \
-DARROW_FLIGHT_SQL=${ARROW_FLIGHT_SQL:-OFF} \
-DARROW_FUZZING=${ARROW_FUZZING:-OFF} \
-DARROW_GANDIVA_PC_CXX_FLAGS=${ARROW_GANDIVA_PC_CXX_FLAGS:-} \
-DARROW_GANDIVA=${ARROW_GANDIVA:-OFF} \
-DARROW_GCS=${ARROW_GCS:-OFF} \
-DARROW_HDFS=${ARROW_HDFS:-ON} \
-DARROW_INSTALL_NAME_RPATH=${ARROW_INSTALL_NAME_RPATH:-ON} \
-DARROW_JEMALLOC=${ARROW_JEMALLOC:-ON} \
-DARROW_JSON=${ARROW_JSON:-ON} \
-DARROW_LARGE_MEMORY_TESTS=${ARROW_LARGE_MEMORY_TESTS:-OFF} \
-DARROW_MIMALLOC=${ARROW_MIMALLOC:-OFF} \
-DARROW_NO_DEPRECATED_API=${ARROW_NO_DEPRECATED_API:-OFF} \
-DARROW_ORC=${ARROW_ORC:-OFF} \
-DARROW_PARQUET=${ARROW_PARQUET:-OFF} \
-DARROW_RUNTIME_SIMD_LEVEL=${ARROW_RUNTIME_SIMD_LEVEL:-MAX} \
-DARROW_S3=${ARROW_S3:-OFF} \
-DARROW_SIMD_LEVEL=${ARROW_SIMD_LEVEL:-DEFAULT} \
-DARROW_SKYHOOK=${ARROW_SKYHOOK:-OFF} \
-DARROW_SUBSTRAIT=${ARROW_SUBSTRAIT:-OFF} \
-DARROW_TEST_LINKAGE=${ARROW_TEST_LINKAGE:-shared} \
-DARROW_TEST_MEMCHECK=${ARROW_TEST_MEMCHECK:-OFF} \
-DARROW_USE_ASAN=${ARROW_USE_ASAN:-OFF} \
-DARROW_USE_CCACHE=${ARROW_USE_CCACHE:-ON} \
-DARROW_USE_GLOG=${ARROW_USE_GLOG:-OFF} \
-DARROW_USE_LD_GOLD=${ARROW_USE_LD_GOLD:-OFF} \
-DARROW_USE_MOLD=${ARROW_USE_MOLD:-OFF} \
-DARROW_USE_PRECOMPILED_HEADERS=${ARROW_USE_PRECOMPILED_HEADERS:-OFF} \
-DARROW_USE_STATIC_CRT=${ARROW_USE_STATIC_CRT:-OFF} \
-DARROW_USE_TSAN=${ARROW_USE_TSAN:-OFF} \
-DARROW_USE_UBSAN=${ARROW_USE_UBSAN:-OFF} \
-DARROW_VERBOSE_THIRDPARTY_BUILD=${ARROW_VERBOSE_THIRDPARTY_BUILD:-OFF} \
-DARROW_WITH_BROTLI=${ARROW_WITH_BROTLI:-OFF} \
-DARROW_WITH_BZ2=${ARROW_WITH_BZ2:-OFF} \
-DARROW_WITH_LZ4=${ARROW_WITH_LZ4:-OFF} \
-DARROW_WITH_OPENTELEMETRY=${ARROW_WITH_OPENTELEMETRY:-OFF} \
-DARROW_WITH_MUSL=${ARROW_WITH_MUSL:-OFF} \
-DARROW_WITH_SNAPPY=${ARROW_WITH_SNAPPY:-OFF} \
-DARROW_WITH_UCX=${ARROW_WITH_UCX:-OFF} \
-DARROW_WITH_UTF8PROC=${ARROW_WITH_UTF8PROC:-ON} \
-DARROW_WITH_ZLIB=${ARROW_WITH_ZLIB:-OFF} \
-DARROW_WITH_ZSTD=${ARROW_WITH_ZSTD:-OFF} \
-DAWSSDK_SOURCE=${AWSSDK_SOURCE:-} \
-DAzure_SOURCE=${Azure_SOURCE:-} \
-Dbenchmark_SOURCE=${benchmark_SOURCE:-} \
-DBOOST_SOURCE=${BOOST_SOURCE:-} \
-DBrotli_SOURCE=${Brotli_SOURCE:-} \
-DBUILD_WARNING_LEVEL=${BUILD_WARNING_LEVEL:-CHECKIN} \
-Dc-ares_SOURCE=${cares_SOURCE:-} \
-DCMAKE_BUILD_TYPE=${ARROW_BUILD_TYPE:-debug} \
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE:-OFF} \
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
-DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-Dgflags_SOURCE=${gflags_SOURCE:-} \
-Dgoogle_cloud_cpp_storage_SOURCE=${google_cloud_cpp_storage_SOURCE:-} \
-DgRPC_SOURCE=${gRPC_SOURCE:-} \
-DGTest_SOURCE=${GTest_SOURCE:-} \
-Dlz4_SOURCE=${lz4_SOURCE:-} \
-DORC_SOURCE=${ORC_SOURCE:-} \
-DPARQUET_BUILD_EXAMPLES=${PARQUET_BUILD_EXAMPLES:-OFF} \
-DPARQUET_BUILD_EXECUTABLES=${PARQUET_BUILD_EXECUTABLES:-OFF} \
-DPARQUET_REQUIRE_ENCRYPTION=${PARQUET_REQUIRE_ENCRYPTION:-ON} \
-DProtobuf_SOURCE=${Protobuf_SOURCE:-} \
-DRapidJSON_SOURCE=${RapidJSON_SOURCE:-} \
-Dre2_SOURCE=${re2_SOURCE:-} \
-DSnappy_SOURCE=${Snappy_SOURCE:-} \
-DThrift_SOURCE=${Thrift_SOURCE:-} \
-Dutf8proc_SOURCE=${utf8proc_SOURCE:-} \
-Dzstd_SOURCE=${zstd_SOURCE:-} \
-Dxsimd_SOURCE=${xsimd_SOURCE:-} \
-G "${CMAKE_GENERATOR:-Ninja}" \
${ARROW_CMAKE_ARGS} \
${source_dir}
if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
if [ "${UBUNTU}" = "20.04" ]; then
echo "arrow emscripten build is not supported on Ubuntu 20.04, run with UBUNTU=22.04"
exit -1
fi
n_jobs=2 # Emscripten build fails on docker unless this is set really low
source ~/emsdk/emsdk_env.sh
emcmake cmake \
--preset=ninja-${ARROW_BUILD_TYPE:-debug}-emscripten \
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE:-OFF} \
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
-DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
${ARROW_CMAKE_ARGS} \
${source_dir}
else
cmake \
-Dabsl_SOURCE=${absl_SOURCE:-} \
-DARROW_ACERO=${ARROW_ACERO:-OFF} \
-DARROW_AZURE=${ARROW_AZURE:-OFF} \
-DARROW_BOOST_USE_SHARED=${ARROW_BOOST_USE_SHARED:-ON} \
-DARROW_BUILD_BENCHMARKS_REFERENCE=${ARROW_BUILD_BENCHMARKS:-OFF} \
-DARROW_BUILD_BENCHMARKS=${ARROW_BUILD_BENCHMARKS:-OFF} \
-DARROW_BUILD_EXAMPLES=${ARROW_BUILD_EXAMPLES:-OFF} \
-DARROW_BUILD_INTEGRATION=${ARROW_BUILD_INTEGRATION:-OFF} \
-DARROW_BUILD_SHARED=${ARROW_BUILD_SHARED:-ON} \
-DARROW_BUILD_STATIC=${ARROW_BUILD_STATIC:-ON} \
-DARROW_BUILD_TESTS=${ARROW_BUILD_TESTS:-OFF} \
-DARROW_BUILD_UTILITIES=${ARROW_BUILD_UTILITIES:-ON} \
-DARROW_COMPUTE=${ARROW_COMPUTE:-ON} \
-DARROW_CSV=${ARROW_CSV:-ON} \
-DARROW_CUDA=${ARROW_CUDA:-OFF} \
-DARROW_CXXFLAGS=${ARROW_CXXFLAGS:-} \
-DARROW_CXX_FLAGS_DEBUG="${ARROW_CXX_FLAGS_DEBUG:-}" \
-DARROW_CXX_FLAGS_RELEASE="${ARROW_CXX_FLAGS_RELEASE:-}" \
-DARROW_CXX_FLAGS_RELWITHDEBINFO="${ARROW_CXX_FLAGS_RELWITHDEBINFO:-}" \
-DARROW_C_FLAGS_DEBUG="${ARROW_C_FLAGS_DEBUG:-}" \
-DARROW_C_FLAGS_RELEASE="${ARROW_C_FLAGS_RELEASE:-}" \
-DARROW_C_FLAGS_RELWITHDEBINFO="${ARROW_C_FLAGS_RELWITHDEBINFO:-}" \
-DARROW_DATASET=${ARROW_DATASET:-OFF} \
-DARROW_DEPENDENCY_SOURCE=${ARROW_DEPENDENCY_SOURCE:-AUTO} \
-DARROW_ENABLE_THREADING=${ARROW_ENABLE_THREADING:-ON} \
-DARROW_ENABLE_TIMING_TESTS=${ARROW_ENABLE_TIMING_TESTS:-ON} \
-DARROW_EXTRA_ERROR_CONTEXT=${ARROW_EXTRA_ERROR_CONTEXT:-OFF} \
-DARROW_FILESYSTEM=${ARROW_FILESYSTEM:-ON} \
-DARROW_FLIGHT=${ARROW_FLIGHT:-OFF} \
-DARROW_FLIGHT_SQL=${ARROW_FLIGHT_SQL:-OFF} \
-DARROW_FUZZING=${ARROW_FUZZING:-OFF} \
-DARROW_GANDIVA_PC_CXX_FLAGS=${ARROW_GANDIVA_PC_CXX_FLAGS:-} \
-DARROW_GANDIVA=${ARROW_GANDIVA:-OFF} \
-DARROW_GCS=${ARROW_GCS:-OFF} \
-DARROW_HDFS=${ARROW_HDFS:-ON} \
-DARROW_INSTALL_NAME_RPATH=${ARROW_INSTALL_NAME_RPATH:-ON} \
-DARROW_JEMALLOC=${ARROW_JEMALLOC:-ON} \
-DARROW_JSON=${ARROW_JSON:-ON} \
-DARROW_LARGE_MEMORY_TESTS=${ARROW_LARGE_MEMORY_TESTS:-OFF} \
-DARROW_MIMALLOC=${ARROW_MIMALLOC:-OFF} \
-DARROW_NO_DEPRECATED_API=${ARROW_NO_DEPRECATED_API:-OFF} \
-DARROW_ORC=${ARROW_ORC:-OFF} \
-DARROW_PARQUET=${ARROW_PARQUET:-OFF} \
-DARROW_RUNTIME_SIMD_LEVEL=${ARROW_RUNTIME_SIMD_LEVEL:-MAX} \
-DARROW_S3=${ARROW_S3:-OFF} \
-DARROW_SIMD_LEVEL=${ARROW_SIMD_LEVEL:-DEFAULT} \
-DARROW_SKYHOOK=${ARROW_SKYHOOK:-OFF} \
-DARROW_SUBSTRAIT=${ARROW_SUBSTRAIT:-OFF} \
-DARROW_TEST_LINKAGE=${ARROW_TEST_LINKAGE:-shared} \
-DARROW_TEST_MEMCHECK=${ARROW_TEST_MEMCHECK:-OFF} \
-DARROW_USE_ASAN=${ARROW_USE_ASAN:-OFF} \
-DARROW_USE_CCACHE=${ARROW_USE_CCACHE:-ON} \
-DARROW_USE_GLOG=${ARROW_USE_GLOG:-OFF} \
-DARROW_USE_LD_GOLD=${ARROW_USE_LD_GOLD:-OFF} \
-DARROW_USE_MOLD=${ARROW_USE_MOLD:-OFF} \
-DARROW_USE_PRECOMPILED_HEADERS=${ARROW_USE_PRECOMPILED_HEADERS:-OFF} \
-DARROW_USE_STATIC_CRT=${ARROW_USE_STATIC_CRT:-OFF} \
-DARROW_USE_TSAN=${ARROW_USE_TSAN:-OFF} \
-DARROW_USE_UBSAN=${ARROW_USE_UBSAN:-OFF} \
-DARROW_VERBOSE_THIRDPARTY_BUILD=${ARROW_VERBOSE_THIRDPARTY_BUILD:-OFF} \
-DARROW_WITH_BROTLI=${ARROW_WITH_BROTLI:-OFF} \
-DARROW_WITH_BZ2=${ARROW_WITH_BZ2:-OFF} \
-DARROW_WITH_LZ4=${ARROW_WITH_LZ4:-OFF} \
-DARROW_WITH_OPENTELEMETRY=${ARROW_WITH_OPENTELEMETRY:-OFF} \
-DARROW_WITH_MUSL=${ARROW_WITH_MUSL:-OFF} \
-DARROW_WITH_SNAPPY=${ARROW_WITH_SNAPPY:-OFF} \
-DARROW_WITH_UCX=${ARROW_WITH_UCX:-OFF} \
-DARROW_WITH_UTF8PROC=${ARROW_WITH_UTF8PROC:-ON} \
-DARROW_WITH_ZLIB=${ARROW_WITH_ZLIB:-OFF} \
-DARROW_WITH_ZSTD=${ARROW_WITH_ZSTD:-OFF} \
-DAWSSDK_SOURCE=${AWSSDK_SOURCE:-} \
-DAzure_SOURCE=${Azure_SOURCE:-} \
-Dbenchmark_SOURCE=${benchmark_SOURCE:-} \
-DBOOST_SOURCE=${BOOST_SOURCE:-} \
-DBrotli_SOURCE=${Brotli_SOURCE:-} \
-DBUILD_WARNING_LEVEL=${BUILD_WARNING_LEVEL:-CHECKIN} \
-Dc-ares_SOURCE=${cares_SOURCE:-} \
-DCMAKE_BUILD_TYPE=${ARROW_BUILD_TYPE:-debug} \
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE:-OFF} \
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
-DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-Dgflags_SOURCE=${gflags_SOURCE:-} \
-Dgoogle_cloud_cpp_storage_SOURCE=${google_cloud_cpp_storage_SOURCE:-} \
-DgRPC_SOURCE=${gRPC_SOURCE:-} \
-DGTest_SOURCE=${GTest_SOURCE:-} \
-Dlz4_SOURCE=${lz4_SOURCE:-} \
-DORC_SOURCE=${ORC_SOURCE:-} \
-DPARQUET_BUILD_EXAMPLES=${PARQUET_BUILD_EXAMPLES:-OFF} \
-DPARQUET_BUILD_EXECUTABLES=${PARQUET_BUILD_EXECUTABLES:-OFF} \
-DPARQUET_REQUIRE_ENCRYPTION=${PARQUET_REQUIRE_ENCRYPTION:-ON} \
-DProtobuf_SOURCE=${Protobuf_SOURCE:-} \
-DRapidJSON_SOURCE=${RapidJSON_SOURCE:-} \
-Dre2_SOURCE=${re2_SOURCE:-} \
-DSnappy_SOURCE=${Snappy_SOURCE:-} \
-DThrift_SOURCE=${Thrift_SOURCE:-} \
-Dutf8proc_SOURCE=${utf8proc_SOURCE:-} \
-Dzstd_SOURCE=${zstd_SOURCE:-} \
-Dxsimd_SOURCE=${xsimd_SOURCE:-} \
-G "${CMAKE_GENERATOR:-Ninja}" \
${ARROW_CMAKE_ARGS} \
${source_dir}
fi

export CMAKE_BUILD_PARALLEL_LEVEL=${CMAKE_BUILD_PARALLEL_LEVEL:-$[${n_jobs} + 1]}
time cmake --build . --target install
Expand Down
4 changes: 4 additions & 0 deletions ci/scripts/cpp_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ case "$(uname)" in
;;
esac

if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
n_jobs=1 # avoid spurious fails on emscripten due to loading too many big executables
fi

pushd ${build_dir}

if [ -z "${PYTHON}" ] && ! which python > /dev/null 2>&1; then
Expand Down
Empty file modified ci/scripts/go_bench.sh
100644 → 100755
Empty file.
46 changes: 45 additions & 1 deletion cpp/CMakePresets.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,32 @@
"CMAKE_BUILD_TYPE": "RelWithDebInfo"
}
},
{
"name": "features-emscripten",
"hidden": true,
"cacheVariables": {
"ARROW_ACERO": "ON",
"ARROW_BUILD_SHARED": "OFF",
"ARROW_BUILD_STATIC": "ON",
"ARROW_CUDA": "OFF",
"ARROW_DEPENDENCY_SOURCE": "BUNDLED",
"ARROW_DEPENDENCY_USE_SHARED": "OFF",
"ARROW_ENABLE_THREADING": "OFF",
"ARROW_FLIGHT": "OFF",
"ARROW_IPC": "ON",
"ARROW_JEMALLOC": "OFF",
"ARROW_MIMALLOC": "OFF",
"ARROW_ORC": "ON",
"ARROW_RUNTIME_SIMD_LEVEL": "NONE",
"ARROW_S3": "OFF",
"ARROW_SIMD_LEVEL": "NONE",
"ARROW_SUBSTRAIT": "ON",
"ARROW_WITH_BROTLI": "ON",
"ARROW_WITH_OPENTELEMETRY": "OFF",
"ARROW_WITH_SNAPPY": "ON",
"CMAKE_C_BYTE_ORDER": "LITTLE_ENDIAN"
}
},
{
"name": "features-minimal",
"hidden": true,
Expand Down Expand Up @@ -341,6 +367,24 @@
"displayName": "Release build with CUDA integration",
"cacheVariables": {}
},
{
"name": "ninja-debug-emscripten",
"inherits": [
"features-emscripten",
"base-debug"
],
"displayName": "Debug build which builds an Emscripten library",
"cacheVariables": {}
},
{
"name": "ninja-release-emscripten",
"inherits": [
"features-emscripten",
"base-release"
],
"displayName": "Release build which builds an Emscripten library",
"cacheVariables": {}
},
{
"name": "ninja-release-flight",
"inherits": [
Expand Down Expand Up @@ -447,4 +491,4 @@
}
}
]
}
}
24 changes: 24 additions & 0 deletions cpp/build-support/emscripten-test-init.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

var Module = {
};

// make sure tests can access the current parquet test data files
Module.preRun = () => {ENV.PARQUET_TEST_DATA = process.env.PARQUET_TEST_DATA;
ENV.ARROW_TEST_DATA = process.env.ARROW_TEST_DATA;
};
4 changes: 2 additions & 2 deletions cpp/cmake_modules/BuildUtils.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -760,8 +760,8 @@ function(ADD_TEST_CASE REL_TEST_NAME)
valgrind --suppressions=valgrind.supp --tool=memcheck --gen-suppressions=all \
--num-callers=500 --leak-check=full --leak-check-heuristics=stdstring \
--error-exitcode=1 ${TEST_PATH} ${ARG_TEST_ARGUMENTS}")
elseif(WIN32)
add_test(${TEST_NAME} ${TEST_PATH} ${ARG_TEST_ARGUMENTS})
elseif(WIN32 OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
add_test(NAME ${TEST_NAME} COMMAND ${TEST_NAME} ${ARG_TEST_ARGUMENTS})
else()
add_test(${TEST_NAME}
${BUILD_SUPPORT_DIR}/run-test.sh
Expand Down
Loading