Skip to content

Commit

Permalink
Refresh and fix SDK Docker image build scripts
Browse files Browse the repository at this point in the history
The way the script is currently it does not work (anymore). I'm aware
of two issues with the current setup:

1. emsdk underneath installs and uses Node JS and the version of Node JS
   is not pinned by the script and isn't tied to the version of the SDK
   used.

   As a result, emsdk moved to a newer version of Node JS than the one
   that was probably used when the scripts were created. This new
   version (it seems, the version currently being used is 18.20.3)
   requires a relatively fresh version of glibc which just isn't
   available in Ubuntu Bionic used as a base image. That results in
   obscure errors from CMake (see
   #170)
2. for whatever reason, the version of the protobuf static libraries
   currently in the repo, don't seem to work. I don't really know
   how the issue happened, but there were at least 2 users that faced
   the problem (see
   #161).

To resolve the first issue, we have a bunch of options:

1. Fix the version of emsdk
2. Fix the version of Node JS
3. Update the version of glibc

I figured I can combine options 1 and 3 together for the following
reasons:

1. Fixing the version of emsdk fixes the problem
2. The problem arised from the fact that the versions of software
   used in the build script are a bit old, so an update might be
   in order, even though we have other solutions to the problem.

> NOTE: It's my understanding that fixing emsdk version should also
> pin Node JS version, so if we deploy option 1, option 2 seem
> redundant.

For the second problem, I think there is only one ultimate solution
- not store binary artifacts in the repository and instead build
them from the sources in the repo.

It seems that in the past there was a concern that building protobuf
libraries takes a long time - it's still certainly the case. However,
I think we can compensate for that in two ways:

1. Drop WAVM - it does not seem like WAVM is still needed (the project
   itself appears to be dead and hasn't had any updates for at least 2
   years), but also one of the comments in
   #158, that
   removed the WAVM from the docs, also suggests that three doesn't
   seem to be a good reason to keep WAVM in the SDK build script.
2. Take advantage of potential hardware threads in the system when
   calling make - most laptops or servers these days have multiple
   cores, so if we have to build protobuf libraries twice, we can
   speed it up by using more cores.

With all those changes made, the build time adds up to something like
32 minutes on my laptop, compared to the 48m without building
protobuf libraries from sources (and without adding -j option to make).
So I think the additional time spent on building protobuf library is
compensated by other changes.

Signed-off-by: Mikhail Krinkin <krinkin.m.u@gmail.com>
  • Loading branch information
krinkinmu committed Oct 1, 2024
1 parent 54167e2 commit fb6af64
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 25 deletions.
4 changes: 2 additions & 2 deletions Dockerfile-sdk
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM ubuntu:bionic
FROM ubuntu:noble

COPY ./sdk_container.sh /
COPY ./build_wasm.sh /
COPY *.cc *.h *.js *.proto Makefile* *.a /sdk/
COPY *.cc *.h *.js *.proto Makefile* /sdk/

RUN ./sdk_container.sh
Binary file removed libprotobuf-lite.a
Binary file not shown.
Binary file removed libprotobuf.a
Binary file not shown.
53 changes: 30 additions & 23 deletions sdk_container.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,20 @@ set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get upgrade -y
apt-get install -y --no-install-recommends apt-utils ca-certificates
apt-get autoremove -y
apt-get clean
apt-get install -y --no-install-recommends software-properties-common apt-transport-https git wget curl pkg-config autoconf autotools-dev automake libtool cmake python zlib1g-dev
apt-get install -y --no-install-recommends ca-certificates git autoconf autotools-dev automake libtool cmake python-is-python3 zlib1g-dev make xz-utils libzstd-dev

# gcc-7
apt-get install -y --no-install-recommends gcc-7 g++-7 cpp-7
export CC=gcc-7
export CXX=g++-7
export CPP=cpp-7
# The specific version of GCC does not actually matter as long as it's confirmed to work.
# That's why we explicitly pin gcc version (in this case to gcc 13) - it's the version
# which was tested to work.
apt-get install -y --no-install-recommends gcc-13 g++-13 cpp-13
export CC=gcc-13
export CXX=g++-13
export CPP=cpp-13

NUM_CPUS=$(nproc)
JOBS=$((NUM_CPUS>1 ? NUM_CPUS-1 : NUM_CPUS))

# get $HOME
cd
Expand All @@ -41,7 +45,7 @@ git checkout v3.9.1
git submodule update --init --recursive
./autogen.sh
./configure
make
make -j $JOBS
make check
make install
cd
Expand All @@ -50,27 +54,30 @@ rm -rf protobuf
# emscripten
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk update-tags
./emsdk install 3.1.7
./emsdk activate 3.1.7
git checkout 3.1.67
./emsdk install --shallow 3.1.67
./emsdk activate 3.1.67
source ./emsdk_env.sh
cd

git clone https://github.com/protocolbuffers/protobuf protobuf-wasm
cd protobuf-wasm
git checkout v3.9.1
git submodule update --init --recursive
./autogen.sh
emconfigure ./configure --disable-shared CXXFLAGS="-O3 -flto"
emmake make -j $JOBS
cd

cp protobuf-wasm/src/.libs/libprotobuf-lite.a /sdk/libprotobuf-lite.a
cp protobuf-wasm/src/.libs/libprotobuf.a /sdk/libprotobuf.a
rm -rf protobuf-wasm

# abseil (optional)
git clone https://github.com/abseil/abseil-cpp
cd abseil-cpp
git checkout 14550beb3b7b97195e483fb74b5efb906395c31e -b Jul302019 # Jul 30 2019
git checkout 4447c7562e3bc702ade25105912dce503f0c4010 -b lts20240722 # Abseil LTS release 20240722.0
emcmake cmake -DCMAKE_CXX_STANDARD=17 "."
emmake make
emmake make -j $JOBS
cd

# WAVM (optional)
apt-get install -y --no-install-recommends llvm-6.0-dev
git clone https://github.com/WAVM/WAVM
cd WAVM
git checkout 1ec06cd202a922015c9041c5ed84f875453c4dc7 -b Oct152019 # Oct 15 2019
cmake "."
make
make install
cd
rm -rf WAVM

0 comments on commit fb6af64

Please sign in to comment.