Repository navigation
Stop shipping NaCl IRT and build it alongside gamelogic instead? #1993
Description
Activity
Well the TLS stuff might be a bit complicated, reading NaCl's
src/untrusted/irt/irt_private_tls.c. But a naive first attempt could be simply excluding that file from the build.The IRT topic is a subset of:
So even if we don't build the native loader, we may want to build the IRT, with
native_clientbeing a submodule.Now the TLS editor…
Host-mode toolchain to build the tls_edit tool which transforms irt_core_raw into irt_core (only actually needed for 32-bit targets)
I was thinking about it some days ago and I thought we can simply ship
tls_editwith the Saigo SDK.My Saigo SDK script already clones the
native_clientrepository to install some special NaCl headers for sandboxed binaries:nacl/nacl_exception.hnacl/nacl_minidump.hnacl/simd.h
The two first ones we copied (in old versions) in the engine repo because NaCl compilers did not ship them, third one we don't use.
So we can also build
tls_editas part of the toolchain, which it is.Note that I started writing CMake code (not even committed) for building the IRT, and I wanted to start doing the same for the TLS editor (I have not started, but that's almost one line if I'm right).
Both would allow integration in Dæmon cmake scripts and in Saigo SDK cmake scripts.
So basically I see things that way:
sel_ldr: fetched/built as part ofnaclruntimein engine external depsirt: built (similar to cgame and sgame) as part of the engine files, or baked in cgame/sgame.tls_edit: fetched/built as part ofsaigosdkin engine external deps
Here is some CMake code to build
tls_edit, it would be callable by thesaigo-nacl-sdkCMake script for inclusion in the Saigo SDK:Obviously the reason to have the IRT in the first place is to save compile time and disk space for those distributing nexe binaries.
I would prefer to keep the cgame/sgame nexe smaller, but we can still build the IRT as part of the engine to be used like we currently do.
Here is my work-in-progress
illwieckz/nacl-tls-editbranch on the Saigo SDK repository, it makes possible to build and shiptls_editas part of the toolchain:Since the Saigo toolchain is already needed to compile the IRT, it means the loader build doesn't need something else than the target toolchain (for
sel_ldr) and the nacl toolchain (for the IRT), removing the need for a third toolchain (the host toolchain fortls_edit).This also means that the Dæmon engine would be able to build the IRT like any other nexe.
Here is my work-in-progress
illwieckz/nacl-tls-editbranch on the Saigo SDK repository, it makes possible to build and shiptls_editas part of the toolchain:Sounds good. That's a reasonable way to untangle the toolchain dependency knot. First you build Saigo + tls_edit, then both the loader and the IRT are unlocked at the same time.
This also means that the Dæmon engine would be able to build the IRT like any other nexe.
Making people who compile Daemon also compile the IRT doesn't seem like a good idea, since firstly it adds a dependency on the NaCl toolchain for building the engine; and more importantly, as we will need to provide a pre-built loader anyway, and the loader and IRT are tightly coupled, it would make more sense to ship them together. I know you have suggested having the end user also build the loader before, but after working on Native Client more it has become abundantly clear that this is a bad idea, because the loader build is extraordinarily sensitive to details of the toolchain. Examples: that the linker script breaks when the compiler adds a new section; that ld.bfd is the only linker you could use at all due to said script, and that with MinGW you will most likely have to use a toolchain with a specific thread-local storage flavor. This would torpedo the portability of Daemon as software that can be built by end users, since with so many configurations it would fail to build, or even build but with its security properties broken.
What would make sense to me is having sel_ldr (+bootstrap) and irt_core built as external_deps targets. (Not sure if it's better as one combined target or two separate. On one hand they are closely coupled and come from the same repository; on the other it would be two separate CMake invocations if we go with CMake, using separate toolchains.)
What would make sense to me is having sel_ldr (+bootstrap) and irt_core built as external_deps targets. (Not sure if it's better as one combined target or two separate. On one hand they are closely coupled and come from the same repository; on the other it would be two separate CMake invocations if we go with CMake, using separate toolchains.)
One thing is that if we build the IRT in the external_deps, then we would need a
native-saigosdkpseudo-package to build the IRT (the same way we havenative-wasmandnative-pkgconfigas host tools for building macOS dependencies), plus thesaigosdkpackage to be shipped to end users.I may add an option to make possible to not download the Saigo SDK but to build it as part of the external_deps. This option would be disabled by default (because building Saigo is very heavy on resources), but that would make possible for someone who wants it to actually rebuilds oneself the complete trusted stack entirely. If we do that, then such user would likely build Saigo twice, the host one, and the target one, even if they are the same. Avoiding that would require to implement (non-) cross-platform detection and use the target
PATHas hostPATH. We made everything inexternal_depsto be cross-compilation first with cleanPATHseparation…To me the best alternative to building the IRT in engine is to download a prebuilt IRT in
external_deps, because building the IRT inexternal_depsbrings the problem of “we need to build/download two saigo toolchains just to build a single external deps archive”.I was assuming that Saigo would be a different download than the rest of external_deps. If we have any other host-mode deps we could expand it to a broader host-dependent deps archive, in addition to our current target-dependent ones. In fact we're kind of hitting that case already with the
tls_editproposal, since that's not really part of Saigo.I do remember that in the past we discussed the opportunity to have two deps packages, one for the toolchain, one for the libs. For example it would have been a solution for shipping an single amd64 PNaCl on both
macos-amd64andmacos-arm64while shipping two packages of libraries for them. Though that need is going to disappear. One thing that would have made the split uncomfortable was that the IRT (can be seen as a lib) was shipped from PNaCl SDK (the toolchain). It was solvable, but annoying.In fact we're kind of hitting that case already with the
tls_editproposal, since that's not really part of Saigo.If we need
tls_editto build a nexe (and the IRT is one) thentls_editshould be part of the “Saigo SDK”. This is not part of Clang, but Saigo is Clang + GNU binutils + Newlib and things like that, the TLS editor is best placed among those tools.The TLS editor is in a similar place as some headers we can find in the
native_clientrepository: they are required to build some sandboxed code, so we better ship all of them with the SDK, whatever their repository origin.While we're at it throwing in more host tools, I've thought about suggesting Breakpad dump_syms + minidump_stackwalk too. This could be interesting for adding
crash_test.pyto CI or integrating symbol generation in the gamelogic build. dump_syms is annoyingly non-universal though: what you can symbolize depends on the host platform. For example on Windows (maybe Mac too) it doesn't know about ELF, so you can't symbolize nexe.While we're at it throwing in more host tools, I've thought about suggesting Breakpad dump_syms + minidump_stackwalk too.
That would be a good idea to have breakpad in
external_deps. If we want to build the IRT inexternal_depsthen we need an host breakpad in the host toolchain and then having it in the host Saigo would be the same. If we want to build the IRT as part of the engine build, shipping a target breakpad inexternal_depsor with the target Saigo is the same (since target Saigo would be part ofexternal_deps.For example on Windows (maybe Mac too) it doesn't know about ELF, so you can't symbolize nexe.
lolwat??? I wonder if we can add some
ifdefto build on/for Windows the ELF breakpad them, and name itnacl-breakpad…Edit: Breakpad using
configureis painful…Yes, the MinGW Breakpad dump_syms_dwarf can be built for Linux too.
We never implemented Mac Breakpad but my suspicion would be that it only symbolizes and is symbolized by itself since it uses yet another executable format. I think besides MinGW, binary format stuff from the host OS is used.
Here is an attempt to build a
dump_syms_elffor Windows with MinGW:Actually it may be a good idea to modify the breakpad build system to generate symbolizers not for the platform but for the format, with names like:
dump_syms_elfdump_syms_dwarf(maybe add some hint it's for Windows, likedwarfpe).dump_syms_macho
It would be awesome to be able to build and run any of them from anywhere.
Actually it's very funny that both
elf,dwarfandmachocan all be used as game/story character/class names 🫣️, this wouldn't work withdwarfpe. 😂️@slipher On Linux we can build both the Linux, macOS and MinGW-DWARF symbolizers (it looks like they are built by default):
$ src/tools/linux/dump_syms/dump_syms Usage: src/tools/linux/dump_syms/dump_syms [OPTION] <binary-with-debugging-info> [directories-for-debug-file] Options: -i: Output module header information only. -c Do not generate CFI section -r Do not handle inter-compilation unit references -v Print all warnings to stderr -n <name> Use specified name for name of the object -o <os> Use specified name for the operating system$ src/tools/mac/dump_syms/dump_syms_mac Must specify Mach-o file Output a Breakpad symbol file from a Mach-o file. Usage: src/tools/mac/dump_syms/dump_syms_mac [-a ARCHITECTURE] [-c] [-g dSYM path] <Mach-o file> -i: Output module header information only. -a: Architecture type [default: native, or whatever is in the file, if it contains only one architecture] -g: Debug symbol file (dSYM) to dump in addition to the Mach-o file -c: Do not generate CFI section -r: Do not handle inter-compilation unit references -h: Usage -?: Usage$ src/tools/windows/dump_syms_dwarf/dump_syms_dwarf Usage: src/tools/windows/dump_syms_dwarf/dump_syms_dwarf [OPTION] <binary-with-dwarf-debugging-info> [directories-for-debug-file] Options: -c Do not generate CFI section -r Do not handle inter-compilation unit referencesIn fact it looks like the Makefile doesn't suppose you build the macOS symbolizer on macOS…
Anyway, with this branch it's now possible to build and run the ELF
dump_symsfor Linux, Windows and macOS:So we will be able to ship the NaCl symbolizer in
external_depsfor every platform to extract nexe symbols as part of the standard nexe build.We decided to keep using a separate, pre-built IRT.
So the
irt_corenexe is untrusted code in the Native Client sandbox implementing mostly the equivalent of libc syscall wrappers for the NaCl VM, plus some other libc stuff like pthreads and malloc. I believe it is treated the same as any other untrusted code by the sandbox. (There are some examples in the NaCl repo'stests/barebones/showing how VM syscalls can be done without the library.) So we might consider ditching the IRT nexe shipped with the engine, and instead build it into each gamelogic binary. This would let us reach the goal of building a Daemon engine without binary blobs much more simply than if we had to buildirt_core. To build everything we need fromnative_clientthe normal way you need to set up 3 toolchains:sel_ldrirt_core_rawtls_edittool which transformsirt_core_rawintoirt_core(only actually needed for 32-bit targets)If we build the IRT files as normal gamelogic files, the latter two toolchains would no longer needed to produce a built-from-scratch Daemon release. I'm assuming that
tls_editis only needed for the case where the code is loaded like a shared library, and not if it's built into the main binary.If my assumptions are correct the steps to do this would be:
-B <irt_core path>flag from thenacl_loaderinvocationObviously the reason to have the IRT in the first place is to save compile time and disk space for those distributing nexe binaries. The
irt_core-amd64.nexewe currently ship is 623K. But Bloaty says 46% of that is debug info. Strip that as our build process normally does, perhaps remove some unused parts while linking, and it's maybe a couple hundred KB. Plus Breakpad debug info for the IRT would be generated in the normal build process.Building an
irt_core_rawwith Saigo single-threadedly took 0.94 s for me excluding build system overhead. (I gotTotal command execution time: 0.938376 secondsusing the Scons--debug=timeoption.)