Skip to content

Use ccache for emscripten library build #25392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Sep 29, 2025
Merged

Use ccache for emscripten library build #25392

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
39d2b25
Try ccache for lib build
dschuff Sep 25, 2025
68d1165
smaller build
dschuff Sep 25, 2025
3bc90a5
absolute directory, print stats, restore cache
dschuff Sep 25, 2025
7e6a05f
use a new key
dschuff Sep 25, 2025
7f32ac3
rename phase
dschuff Sep 25, 2025
4312b65
fix env var
dschuff Sep 25, 2025
7ae8ffd
debug ccache
dschuff Sep 25, 2025
6fe5544
unset CCACHE_DIR
dschuff Sep 25, 2025
8156345
name restore step
dschuff Sep 25, 2025
010f7a2
bigger test
dschuff Sep 25, 2025
98ffe31
rename
dschuff Sep 25, 2025
54d3efc
restore all steps
dschuff Sep 25, 2025
cf6fe42
cache based on clang version
dschuff Sep 25, 2025
eaf7d42
remove extraneous change
dschuff Sep 25, 2025
588b0f8
zero the stats before persisting the cache
dschuff Sep 26, 2025
be30413
conditionalize main vs PR
dschuff Sep 26, 2025
50466f5
do a minimal build
dschuff Sep 26, 2025
a408894
try it with named branch
dschuff Sep 26, 2025
0ec58b1
try with repo match
dschuff Sep 26, 2025
e22c0f2
update condition
dschuff Sep 26, 2025
831178e
switch event
dschuff Sep 26, 2025
6ee7b08
use name
dschuff Sep 26, 2025
9a5f05a
undo temporary changes
dschuff Sep 26, 2025
048bdf4
Merge branch 'main' into ccache-libs
dschuff Sep 26, 2025
1ec1225
update condition
dschuff Sep 26, 2025
4f7de3d
use PR_NUMBER
dschuff Sep 26, 2025
c79c297
use not-main
dschuff Sep 26, 2025
1756551
remove debug prints
dschuff Sep 26, 2025
e9ad070
fix posixtestsuite
dschuff Sep 26, 2025
dfb4244
remove unnecessary condition, and add comment
dschuff Sep 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 28 additions & 21 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,31 +169,31 @@ commands:
command: |
./emcc --clear-cache
- pip-install
- run: apt-get install -q -y ninja-build
- run: apt-get install -q -y ninja-build ccache
- run:
name: embuilder build ALL
command: |
./embuilder build ALL
./test/runner test_hello_world
- run:
name: embuilder (LTO)
command: |
./embuilder build MINIMAL --lto
./test/runner test_hello_world
- run:
name: embuilder (WASM64)
command: |
./embuilder build MINIMAL --wasm64
- run:
name: embuilder (PIC)
command: |
./embuilder build MINIMAL_PIC --pic
./test/runner test_hello_world
- run:
name: embuilder (PIC+LTO)
command: |
./embuilder build MINIMAL --pic --lto
./embuilder build MINIMAL
./test/runner test_hello_world
# - run:
# name: embuilder (LTO)
# command: |
# ./embuilder build MINIMAL --lto
# ./test/runner test_hello_world
# - run:
# name: embuilder (WASM64)
# command: |
# ./embuilder build MINIMAL --wasm64
# - run:
# name: embuilder (PIC)
# command: |
# ./embuilder build MINIMAL_PIC --pic
# ./test/runner test_hello_world
# - run:
# name: embuilder (PIC+LTO)
# command: |
# ./embuilder build MINIMAL --pic --lto
# ./test/runner test_hello_world
persist:
description: "Persist the emsdk, libraries, and engines"
steps:
Expand All @@ -207,6 +207,11 @@ commands:
- vms/
- wasi-sdk/
- .jsvu/
- save_cache:
paths:
- ccache_dir/
key: "ccache"

prepare-for-tests:
description: "Setup emscripten tests"
steps:
Expand Down Expand Up @@ -535,6 +540,8 @@ jobs:
environment:
EMCC_CORES: 16
EMCC_USE_NINJA: 1
CCACHE_BASE_DIR: "ccache_dir"
EM_COMPILER_WRAPPER: "ccache"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is enough. I know that @juj had a custom patch for ccache and added a specific _EMCC_CCACHE feature to support it. See #13681

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I saw that but I don't really see why it's necessary. I figured I'd try the easy thing first, and if it looks like it works, dig more into that. It may be that since we just want to use it for library compilation, it's enough.

@juj can you say more about why you wanted ccache to wrap the entire emscripten driver rather than just the underlying clang?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @dschuff is trying to integrate ccache in another way: in the backend between emcc and clang.

My emsdk support is placed in user -> ccache -> emcc -> clang. This looks like is doing user -> emcc -> ccache -> clang.

Would be fantastic to see what the performance difference of this approach ends up being.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance-wise, the main difference would obviously be that we still have to run all the python code of the driver. Since this is a compile and not a link, there's not a huge amount of stuff that gets done, but certainly there's a little cost. Probably the builtin profiling support could estimate how much. But mostly I just picked this way because I didn't want to bother with a fork of ccache.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason was performance. Then it would also work for final link, and e.g. wrap over binaryen invocations of wasm-opt and so on.

However I have to say that in my approach, I found performance gains to be very small, so we didn't end up deploying it at Unity. I do have an itch to re-try though, and see if I could optimize the ccache implementation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, @juj, we are trying this out as way to speed up our CI here in circleci. Currently there is no caching so each needs to build everything from scratch.

We are looking at some kind of shared cache in combination with heuristic_clear_cache.py, or maybe just relying on ccache to notice when llvm changes.

BTW, how do your building handle LLVM changes? I guess you somehow clobber the build when llvm changes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I was under the impression that ccache doesn't work for linking or other cases that aren't basically just source -> object file. It just falls back to the underlying compiler.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, how do your building handle LLVM changes?

My ccache port looked at git hash if it was detected (developer installation), or if not, then emscripten-version.txt.

Additionally the contents of EM_CONFIG was hashed into the state.

The implementation can be seen here: ccache/ccache@master...juj:ccache:emscripten

Ah, I was under the impression that ccache doesn't work for linking or other cases that aren't basically just source -> object file. It just falls back to the underlying compiler.

Err actually, now that I scan through my fork, I think you are right: ccache does not cache link commands, only compile commands. So my fork didn't end up helping cache any wasm-opt calls either.

I haven't worked on the ccache fork in a while now - it looks like something has changed in CMake that it does not build with CMake >= 4 anymore. So it would need some freshening to bring up to speed again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My plan is to just key on the git version of clang or the emsdk installed by CI. Since I'm just trying to cache clang's output and not emscripten's, I think everything should be included in the clang version, the flags, the file inputs (i.e. the headers and sources).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't currently actually know whether ccache automatically takes the compiler version into account or not, but in order to have CircleCI automatically save and restore the cache across builds, I have to give it a cache key anyway.

steps:
- checkout
- run:
Expand Down
2 changes: 1 addition & 1 deletion test/third_party/posixtestsuite
Open in desktop
Submodule posixtestsuite updated 5 files
+0 −20 conformance/interfaces/pthread_cancel/5-2.c
+1 −13 conformance/interfaces/pthread_cond_init/4-2.c
+0 −11 conformance/interfaces/pthread_kill/8-1.c
+0 −12 conformance/interfaces/pthread_mutex_init/5-3.c
+1 −13 conformance/interfaces/pthread_mutex_lock/3-1.c
4 changes: 2 additions & 2 deletions tools/compile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ def array_contains_any_of(hay, needles):

if '-nostdinc' not in user_args:
if not settings.USE_SDL:
cflags += ['-Xclang', '-iwithsysroot' + os.path.join('/include', 'fakesdl')]
cflags += ['-Xclang', '-iwithsysroot' + os.path.join('/include', 'compat')]
cflags += ['-Xclang','-iwithsysroot' + os.path.join('/include', 'fakesdl')]
cflags += ['-Xclang','-iwithsysroot' + os.path.join('/include', 'compat')]

return cflags