-
Notifications
You must be signed in to change notification settings - Fork 3.7k
GH-23221: [C++] Add support for building with Emscripten #37821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
110 commits
Select commit
Hold shift + click to select a range
5503984
cmake changes for emscripten
joemarshall 331aaa3
reverted cmakelists from python
joemarshall 3e4156e
documentation tidying after review
joemarshall 80b21d1
preset fixes
joemarshall 908abf9
updates to simplify emscripten build process
joemarshall 6effae5
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 31ca7f9
build fixes
joemarshall c26d68c
better comments
joemarshall f5ec444
cmake fixes for python emscripten build to work
joemarshall 3c2f179
fix pyodide stdc version number
joemarshall aec8a79
made substrait, ipc, brotli build on emscripten
joemarshall 0db01d7
docs fix
joemarshall 0daa670
bad if statement fix
joemarshall ff58481
cmake-format
joemarshall 008b3dc
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 20f9f05
removed python cmake changes from emscripten C++ build pr
joemarshall b777f69
documentation fix
joemarshall d93d8cc
preset fixes
joemarshall 831715d
toolchain fixes
joemarshall 3fd8a1c
typo
joemarshall 28f675d
fix for webassembly where size_t != int64_t
joemarshall 97e58f9
size_t not always == in64_t
joemarshall a00f068
reverting size_t fixes
joemarshall bda2b26
debug build fixes
joemarshall 14b4968
fixed order of build preset dependencies
joemarshall 834b5a3
debug build fixes
joemarshall ff2a7c9
cmake format
joemarshall 45ece1e
typos etc. from code review
joemarshall 28ccba9
review fixes
joemarshall 7a8d71a
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall adc68e8
docs fixes
joemarshall 7cb0da9
comments grammar
joemarshall e3f1a81
lint
joemarshall 123813f
correct separators in externalproject call
joemarshall 9fff2ce
test changes for emscripten
joemarshall 0b5cc3b
emscripten archery fixes
joemarshall b5fedaa
force non-parallel emscripten debug build
joemarshall 76d464b
split dwarf debug info on emscripten
joemarshall b2dd85b
use limited debug info on emscripten builds
joemarshall 8d3b255
disable test that won't work on emscripten
joemarshall 8b7d2e4
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 84867d2
set debug flag to g2
joemarshall 9c7ad2e
lint
joemarshall 275c08a
typo
joemarshall 22ace85
test fixes
joemarshall 527e477
fixes to tests for emscripten
joemarshall a96f793
emscripten test fixes
joemarshall 5e8ee70
more test fixes
joemarshall ae1c579
fix to tests for emscripten
joemarshall 688d1ab
lint
joemarshall da548e6
formatting
joemarshall b457d7c
dockerfile fixes in code review
joemarshall 92ead7c
review updates
joemarshall 2a1d67f
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall 58fbdfe
cmake-format updated version
joemarshall e9e1988
lint
joemarshall ddbcf56
licence on test init js
joemarshall 1d4cc2c
fix to preprocessor directive check
joemarshall e3631fd
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 8b8f8d0
fix #defines
joemarshall 63d497a
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 6124d53
review fixes
joemarshall b10b415
use __EMSCRIPTEN__ because emscripten docs say so
joemarshall 09bffa0
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall c9a79cc
whitespace
joemarshall 4980d48
build fixes after review
joemarshall d6026f9
fix zlib on wasm
joemarshall fed5439
fix to csv test
joemarshall 48c795b
typo #if! instead of ifndef
joemarshall 4f2ebfb
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall 7b01717
revert test changes
joemarshall f780784
include config.h for threading define
joemarshall d23f1a1
lint
joemarshall bdd051d
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall f79e49f
dead code elimination in test executables
joemarshall 232d7f5
revert main_module change
joemarshall 5b65a36
default to -O1 for emscripten
joemarshall 72481cf
lint
joemarshall 57e1b65
Apply suggestions from code review
joemarshall 9506ca5
use -sRELOCATABLE instead of -fPIC
joemarshall 617ba51
remove emscriptenoverrides
joemarshall dd2c662
review changes
joemarshall b1050f5
Update cpp/src/arrow/util/io_util.cc
joemarshall 145426f
restrict number of concurrent processes in emscripten
joemarshall 064659c
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall 8cc5ae3
lint fixes
joemarshall 2429261
Apply suggestions from code review
joemarshall 370b1ff
Apply suggestions from code review
joemarshall d0fa881
Apply suggestions from code review
joemarshall da176ad
documentation consistency
joemarshall 5f058c9
don't ignore conversion errors any more
joemarshall 51705de
build fixes from review
joemarshall 225cef4
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall 3179e57
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall d3e0770
typo
joemarshall abcd7df
fix script permissions
joemarshall cc0c1d5
toolchain lint
joemarshall 701d5af
test commit:
joemarshall 0064fa0
reverted test commit
joemarshall 91a6e36
Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
joemarshall 07f2268
Merge branch 'emscripten_cmake_changes' of github.com:joemarshall/arr…
joemarshall 028682e
Apply suggestions from code review
joemarshall d36fca2
code review changes
joemarshall ee99e5e
toolchain build fix
joemarshall b73740a
cmake fixes for emscripten
joemarshall 4dc3d03
alphabetical order oops
joemarshall 8b07ebf
remove dynamic linking test on emsripten
joemarshall c0692da
lint
joemarshall d3bc3b4
Use arrow/util/config.h
kou 1307910
Remove trailing spaces
kou File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -65,6 +65,7 @@ RUN latest_system_llvm=14 && \ | |
RUN apt-get update -y -q && \ | ||
apt-get install -y -q --no-install-recommends \ | ||
autoconf \ | ||
bzip2 \ | ||
ca-certificates \ | ||
ccache \ | ||
cmake \ | ||
|
@@ -115,10 +116,20 @@ RUN apt-get update -y -q && \ | |
rapidjson-dev \ | ||
rsync \ | ||
tzdata \ | ||
wget && \ | ||
wget \ | ||
xz-utils && \ | ||
apt-get clean && \ | ||
rm -rf /var/lib/apt/lists* | ||
|
||
# install emscripten using EMSDK | ||
ARG emscripten_version="3.1.45" | ||
RUN cd ~ && git clone https://github.com/emscripten-core/emsdk.git && \ | ||
cd emsdk && \ | ||
./emsdk install ${emscripten_version} && \ | ||
./emsdk activate ${emscripten_version} && \ | ||
echo "Installed emsdk to:" ~/emsdk | ||
|
||
|
||
ARG gcc_version="" | ||
RUN if [ "${gcc_version}" = "" ]; then \ | ||
apt-get update -y -q && \ | ||
|
@@ -151,6 +162,9 @@ RUN if [ "${gcc_version}" = "" ]; then \ | |
update-alternatives --set c++ /usr/bin/g++; \ | ||
fi | ||
|
||
# make sure zlib is cached in the EMSDK folder | ||
RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any reason why this isn't together with the install step above? |
||
|
||
COPY ci/scripts/install_minio.sh /arrow/ci/scripts/ | ||
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local | ||
|
||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
// Licensed to the Apache Software Foundation (ASF) under one | ||
// or more contributor license agreements. See the NOTICE file | ||
// distributed with this work for additional information | ||
// regarding copyright ownership. The ASF licenses this file | ||
// to you under the Apache License, Version 2.0 (the | ||
// "License"); you may not use this file except in compliance | ||
// with the License. You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, | ||
// software distributed under the License is distributed on an | ||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
// KIND, either express or implied. See the License for the | ||
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
var Module = { | ||
}; | ||
|
||
// make sure tests can access the current parquet test data files | ||
Module.preRun = () => {ENV.PARQUET_TEST_DATA = process.env.PARQUET_TEST_DATA; | ||
ENV.ARROW_TEST_DATA = process.env.ARROW_TEST_DATA; | ||
}; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, two things: