Be more selective on symbols exported from the kernel

[nyh]

Right now, we export all symbols from the kernel, and they can all be used by shared objects.

This has a number of negative implications.

First is of course that applications can accidentally use kernel objects which we never intended to expose.

The second is that all these symbols (and C++ symbols have L-O-N-G names...) appear in the .dynstr and .dynsym sections of the kernel (loader-stripped.elf), which amounts to almost 1MB.

We should consider only exporting some of the symbols, not all of them.

[efpiva]

I was looking at the visibility suport at G++ and it looks like it can accomplish that.

http://gcc.gnu.org/wiki/Visibility

I can try to work on this, but I would need to know which symbols we need to keep, which aren't important.

Do you have any tip on how to have this list?

Do you think this is the right way to go?

[glommer]

On Fri, Jan 17, 2014 at 4:21 PM, efpiva notifications@github.com wrote:

I was looking at the visibility suport at G++ and it looks like it can
accomplish that.

http://gcc.gnu.org/wiki/Visibility

I can try to work on this, but I would need to know which symbols we need
to keep, which aren't important.

Do you have any tip on how to have this list?

Do you think this is the right way to go?

It's not clear for from me from that description if the "compilation unit"
is the individual .cc file or the whole resulting ELF.

If the former, this scheme won't work: we need symbols defined in one file
to be visible to other files so long as they are part
of the kernel. What we want is to discriminate which symbols can and cannot
be used by the applications on top of it.

If the later, then this may work beautifully. Can you verify that ?

Also, don't worry too much about which symbols. New symbols appear everyday
anyway, so this is and will always be an
ongoing effort. Focus on the mechanism and start with the obvious (linux
syscalls from the compatibility layer), things used by
our in-tree .so's like tests and cpio, mkfs, java, etc.

We can fine tune this later.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97#issuecomment-32601356
.

[efpiva]

I think it's on the result ELF. I'm not sure if what I did on my small test
can be applied the way the ELF is compiled in the makefile. I'll try
latter, but if someone can give me a hint I would apreciate.

I did a small Hello World splitted on two .cc files, the second with 2
functions (helloWorld and helloWorldSub). This is the result:

without the hidden G++ flag.

$ g++ -fPIC -c helloWorld.cc
$ g++ -fPIC -c helloWorldFunction.cc
$ g++ -shared -o hello _.o
$ nm -C -D hello
0000000000201050 B bss_start
U __cxa_atexit
w __cxa_finalize
0000000000201050 D _edata
0000000000201060 B _end
0000000000000a4c T _fini
w __gmon_start
00000000000007c8 T _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
w Jv_RegisterClasses
00000000000009dc T main
0000000000000974 T helloWorld()
0000000000000958 T helloWorldSub()
U std::ios_base::Init::Init()
U std::ios_base::Init::~Init()
U std::cout
U std::basic_ostream<char, std::char_traits >&
std::operator<< std::char_traits(std::basic_ostream<char,
std::char_traits >&, char const)
$ rm *.o
$ rm hello

with the hidden G++ flag.

$ g++ -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -c
helloWorldFunction.cc
$ g++ -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -c helloWorld.cc
$ g++ -fvisibility=hidden -fvisibility-inlines-hidden -shared -o hello _.o
$ nm -C -D hello
0000000000201040 B bss_start
U __cxa_atexit
w __cxa_finalize
0000000000201040 D _edata
0000000000201050 B _end
000000000000096c T _fini
w __gmon_start
0000000000000710 T _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
w Jv_RegisterClasses
U std::ios_base::Init::Init()
U std::ios_base::Init::~Init()
U std::cout
U std::basic_ostream<char, std::char_traits >&
std::operator<< std::char_traits(std::basic_ostream<char,
std::char_traits >&, char const)

On Fri, Jan 17, 2014 at 10:34 AM, Glauber Costa notifications@github.comwrote:

On Fri, Jan 17, 2014 at 4:21 PM, efpiva notifications@github.com wrote:

I was looking at the visibility suport at G++ and it looks like it can
accomplish that.

http://gcc.gnu.org/wiki/Visibility

I can try to work on this, but I would need to know which symbols we
need
to keep, which aren't important.

Do you have any tip on how to have this list?

Do you think this is the right way to go?

It's not clear for from me from that description if the "compilation unit"
is the individual .cc file or the whole resulting ELF.

If the former, this scheme won't work: we need symbols defined in one file
to be visible to other files so long as they are part
of the kernel. What we want is to discriminate which symbols can and
cannot
be used by the applications on top of it.

If the later, then this may work beautifully. Can you verify that ?

Also, don't worry too much about which symbols. New symbols appear
everyday
anyway, so this is and will always be an
ongoing effort. Focus on the mechanism and start with the obvious (linux
syscalls from the compatibility layer), things used by
our in-tree .so's like tests and cpio, mkfs, java, etc.

We can fine tune this later.

—
Reply to this email directly or view it on GitHub<
https://github.com/cloudius-systems/osv/issues/97#issuecomment-32601356>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97#issuecomment-32602124
.

Eduardo Fernandes Piva
efpiva@gmail.com

[nyh]

On Fri, Jan 17, 2014 at 2:34 PM, Glauber Costa notifications@github.comwrote:

It's not clear for from me from that description if the "compilation unit"
is the individual .cc file or the whole resulting ELF.

As far as I know, it is possible to tell the linker to treat all symbols,
by default, as hidden, and in turn make visible specific symbols - either
by giving it a list of symbols to export, or by marking them in the source
code with the "visibility" attribute.

Also, don't worry too much about which symbols. New symbols appear
everyday
anyway, so this is and will always be an
ongoing effort. Focus on the mechanism

I agree - we need to first try to see how this actually works on some
example symbol. Will we use ld options (which), attributes, or what? Will
we need to change our own elf.cc code to ensure that invisible symbols in
the main executable (the kernel) are invisible to the shared objects? I
think the answer to the last question is, luckily, no - we already saw an
example of an invisible symbol in the kernel (with stack unwinding) which
became hidden from shared objects.

and start with the obvious (linux
syscalls from the compatibility layer), things used by
our in-tree .so's like tests and cpio, mkfs, java, etc.

While I agree with this direction in the long term (I'm the one who created
this bug in the bug tracker...), I think it's going to give us a lot of
headaches in the short term. In the past, OSv couldn't run certain
applications because we were missing certain glibc functions. So we added
them one by one. Now, we'll start the same process again, discovering that
symbols that in fact exist, are invisible.

I think it would be interesting to try to think of a mechanism which can
help us more-automatically white-list all the glibc ABI. Perhaps we can get
a list of functions to whitelist (make visible) from "nm -D" of glibc
(questionable whether this is legal?) or by some sort of grep/awk/whatever
on our include/api directory, looking for extern "C" symbols defined there

• and should all be visible.

We'll have an additional annoyance of deciding which non-Linux OSv APIs
should be "public". Since OSv is young and the number of OSv-specific
applications is very small, we don't know what the next one will want to
use.

So I think that solving this issue will give us a lot of headaches, but in
the long term, when OSv becomes mature, it will be necessary.

[glommer]

On Fri, Jan 17, 2014 at 5:40 PM, nyh notifications@github.com wrote:

On Fri, Jan 17, 2014 at 2:34 PM, Glauber Costa <notifications@github.com

wrote:

It's not clear for from me from that description if the "compilation
unit"
is the individual .cc file or the whole resulting ELF.

As far as I know, it is possible to tell the linker to treat all symbols,
by default, as hidden, and in turn make visible specific symbols - either
by giving it a list of symbols to export, or by marking them in the source
code with the "visibility" attribute.

Also, don't worry too much about which symbols. New symbols appear
everyday
anyway, so this is and will always be an
ongoing effort. Focus on the mechanism

I agree - we need to first try to see how this actually works on some
example symbol. Will we use ld options (which), attributes, or what? Will
we need to change our own elf.cc code to ensure that invisible symbols in
the main executable (the kernel) are invisible to the shared objects? I
think the answer to the last question is, luckily, no - we already saw an
example of an invisible symbol in the kernel (with stack unwinding) which
became hidden from shared objects.

and start with the obvious (linux
syscalls from the compatibility layer), things used by
our in-tree .so's like tests and cpio, mkfs, java, etc.

While I agree with this direction in the long term (I'm the one who created
this bug in the bug tracker...), I think it's going to give us a lot of
headaches in the short term. In the past, OSv couldn't run certain
applications because we were missing certain glibc functions. So we added
them one by one. Now, we'll start the same process again, discovering that
symbols that in fact exist, are invisible.

Nadav, you are right in principal and generally, but let me try to bring
some points to the
discussion:

As we have added those symbols, they were added in a fairly localized
fashion. So we
won't have to hunt them. For instance, by default, we can whitelist
everything under libc/.

So I don't think it would be that bad. But of course, having something that
is guaranteed
to get it right every time is better - if we can come up with that.

[efpiva]

Coming back to this issue, actually -fvisibility=hidden do what we need.

The kernel compile successfully, so all symbols can see each other and anything can be linked with anything. The symbols are messed up just at the final linkage stage.

So, during build, I got this error (as expected):
/home/eduardo/workspace/osv/scripts/mkzfs.py -o bare.img -d bare.img.d -m /home/eduardo/workspace/osv/build/release/bootfs.manifest
OSv v0.05-348-g8b39f8c
Failed looking up symbol stderr

And, the final ELF stripped size after this change is 10432656 bytes, before it was 11006096, so it has reduced by half Mb. Another benefit that the GCC wiki says is that, when symbols are supposed to be hidden, some optimization can happen on compile time, so we may have a better performance also. Off course this I can check after all symbols that should be market as visible are correct.

One thing that I was thinking to do is to create a macro called OSV_API, that applies the visibility hing that GCC needs, and apply that macro to all needed API. One thing that would be nice is to enforce during build time that all OSV_API had some sort of documentation, like Doxygen, so we could extract (and publish) in a easy way all APIs that OSv supports (specially for those that are exclusive for OSv), so people don't need to actually read OSv source code to find them out.

[wkozaczuk]

Here is what I propose to identify symbols that need to stay exported/public from kernel (somewhat inspired by trying to solve #1040) and use this code in core/elf.cc as a basis:

    static const auto supplied_modules = {
          "libresolv.so.2",
          "libc.so.6",
          "libm.so.6",
#ifdef __x86_64__
          "ld-linux-x86-64.so.2",
          "libboost_system.so.1.55.0",
          "libboost_program_options.so.1.55.0",
#endif /* __x86_64__ */
#ifdef __aarch64__
          "ld-linux-aarch64.so.1",
          "libboost_system-mt.so.1.55.0",
          "libboost_program_options-mt.so.1.55.0",
#endif /* __aarch64__ */
          "libpthread.so.0",
          "libdl.so.2",
          "librt.so.1",
          "libstdc++.so.6",
          "libaio.so.1",
          "libxenstore.so.3.0",
          "libcrypt.so.1",
    };

1. Find the list of exported symbols by Linux equivalent (nm -CD /lib/x86_64-linux-gnu/libc.so.6) for each library (except the boost ones which we plan to hide anyway).
2. Find the intersection between the sum of the above and symbols exported by OSv (nm -CD build/release/loader-stripped.elf).
3. Use the result of the 2 to add attribute ((visibility ("default"))) in all identified places (provided all sources are compiled with -fvisibility=hidden.

The step 3 might be pretty tedious but maybe we can ease the pain by compiling groups of source files (all directories as is) like musl with NO -fvisibility=hidden.

[wkozaczuk]

Based on this take from the GCC Visibility article:

`To aid you converting old code to use the new system, GCC now supports also a #pragma GCC visibility command:

extern void foo(int);
#pragma GCC visibility push(hidden)
extern void someprivatefunct(int);
#pragma GCC visibility pop
#pragma GCC visibility is stronger than -fvisibility; it affects extern declarations as well. -fvisibility only affects definitions, so that existing code can be recompiled with minimal changes. This is more true for C than C++; C++ interfaces tend use classes, which are affected by -fvisibility.'`

it seems we should be able to put attribute ((visibility ("default"))) in some kind of master header file with externs that would be included everywhere. That way we might not need to manually modify hundreds of files, no?

[nyh]

I think we don't have to play (just yet) with the visibility of all the OSV symbols, and could perhaps just surround the #include of boost headers with a pragma push visibility hidden. I can't check this now, I'm standing in line in Zurich airport. So good luck :-)

[wkozaczuk]

Some critical code snippets intended to make it easier to understand my proposal:

• exported libraries - all C symbols provided to regular Linux apps (vs internal apps like cpiod) belong to one of the libraries listed as part of the supplied_modules array below

    // Our kernel already supplies the features of a bunch of traditional
    // shared libraries:
    static const auto supplied_modules = {
          "libresolv.so.2",
          "libc.so.6",
          "libm.so.6",
#ifdef __x86_64__
          "ld-linux-x86-64.so.2",
          "libc.musl-x86_64.so.1",
          // As noted in issue #1040 Boost version 1.69.0 and above is
          // compiled with hidden visibility, so even if the kernel uses
          // this library, it will not be visible for the application and
          // it will need to load its own version of this library.
#if BOOST_VERSION < 106900
          "libboost_system.so.1.55.0",
#endif
#endif /* __x86_64__ */
#ifdef __aarch64__
          "ld-linux-aarch64.so.1",
#if BOOST_VERSION < 106900
          "libboost_system-mt.so.1.55.0",
#endif
#endif /* __aarch64__ */
          "libpthread.so.0",
          "libdl.so.2",
          "librt.so.1",
          "libstdc++.so.6",
          "libaio.so.1",
          "libxenstore.so.3.0",
          "libcrypt.so.1",
    };

• linked static libraries - all non-hidden symbols in the libraries linked statically as part of the kernel are also implicitly exposed; please note that 3 libraries - libstdc++.a, libgcc_eh.a and boost-libs are linked --whole-archive which means all public symbols (whether used or not) from those libraries are implicitly exposed as well.

$(out)/kernel.elf: $(stage1_targets) arch/$(arch)/loader.ld $(out)/empty_bootfs.o $(loader_options_dep)
	$(call quiet, $(LD) -o $@ --defsym=OSV_KERNEL_BASE=$(kernel_base) \
	    --defsym=OSV_KERNEL_VM_BASE=$(kernel_vm_base) --defsym=OSV_KERNEL_VM_SHIFT=$(kernel_vm_shift) \
		-Bdynamic --export-dynamic --eh-frame-hdr --enable-new-dtags -L$(out)/arch/$(arch) \
	    $(^:%.ld=-T %.ld) \
	    --whole-archive \
	      $(libstdc++.a) $(libgcc_eh.a) \
	      $(boost-libs) \
	    --no-whole-archive $(libgcc.a), \
		LINK kernel.elf)

The initial attempt (or at least one of variants of it, see above) to hide most symbols envisioned compiling all sources with the flags -fvisibility=hidden -fvisibility-inlines-hidden and explicitly annotating the symbols we want to expose with __attribute__ ((visibility ("default"))). Unfortunately, annotating all symbols (possibly over 1K) in the right places would be a pretty painful and tedious exercise (also what about musl sources?). To make things worse, my tests of compiling everything with -fvisibility=hidden -fvisibility-inlines-hidden and even linking libstdc++.a as --no-whole-archive, would leave around 5K symbols still left in kernel.elf (or loader.elf) out of 17K - most of those would be public symbols of libstdc++.a. Now, compiling with -fvisibility=hidden -fvisibility-inlines-hidden should in theory also produce more optimized code due to smaller GOT/PLT (relative jumps) and better inlining, however, as I read and understand https://gcc.gnu.org/wiki/Visibility, those benefits would mostly apply to shared libraries only or PIEs. OSv kernel, on the other hand, is a position-dependent executable with a very small number (28) of symbols to be relocated.

The solution I am proposing below relies mostly on the version script mechanism (the linker flag -version-script) which allows specifying an explicit list of symbols to be exported. The version script mechanism is not documented very well but based on my experiments it allows symbols to be listed more than one time and symbols not found in the source object files are ignored by the linker. The version script mechanism is applied at the linking stage only, so it will not optimize the code as the compiler flags -fvisibility=hidden -fvisibility-inlines-hidden could, but for the reasons spelled out in the paragraph above it does not seem it would in our case. Also if we wanted to (because we thought it would produce better (faster?) code), we could still use those compiler options and annotation as an ancillary mechanism.

So here is a fairly comprehensive recipe of how we can hide most symbols and expose only those needed.

1. Hide standard C++ library. This involves removing libstdc++.so.6 from supplied_modules array in core/elf.cc, linking libstdc++.a in --no-whole-archive mode and modifying manifest files and relevant scripts to add libstdc++.so.6 to the image if given app needs it.
2. Hide libboost_system.so even for boost < 1.69. Also, link the boost system static library in --no-whole-archive mode.
3. Link the libgcc_eh.a library in --no-whole-archive mode - any symbols needs by apps should come from libgcc_s.so if needed by given app and then added to the image (we already add libgcc_s.so).
4. For each "exposed" library by supplied_modules in elf.cc produce a list of symbols that OSv kernel implements and would provide. This can be done by intersecting the list of symbols provided by original so file (for example libc.so.6) on given Linux host (Fedora or Ubuntu) with the list of symbols provided by current kernel.elf/loader.elf. These files could be in the version script format and would be stored in as part of the source tree and then manually maintained if any new symbols were added to the kernel later. The linking step of kernel.elf/`loader.elf``in the makefile would use a version script file that would be concatenated in the fly out of these individual files. Also, we could use the same individual version files to help generate documentation (doxygen?) of the OSv Linux ABI.
5. Expose any C++ symbols used by OSv internal apps/modules like cpiod, httpserver, cloud-init as well as tests as C symbols (we already have core/osv_c_wrappers.cc file for that). These would include options parsing functions from core/options.cc, application::run, and probably some other ones (10-20?) (the unit tests may rely on many more C++ symbols that would also need to be converted (hopefully not)). The corresponding new C symbols would also be added to a separate version script file that would be concatenated along with Linux libraries' ones from the previous step. Ideally, those symbols should only be visible to the internal apps and unit tests, but I am not sure we can achieve it in our dynamic linker logic (we would need to know 'who is asking for the symbol). Finally, all relevant internal apps would either need to be linked statically with C++ standard library, rely on libstdc++.so` added to the image, or converted to C (:-(). So in essence we would only export plain C symbols and hide C++ completely.
6. The steps above should not only hide most symbols but also quite radically shrink kernel size by shrinking the .dynstr and .dynsym sections. But to take full advantage of hiding most symbols and shrink the kernel, even more, we would also compile all sources with the -ffunction-sections -fdata-sections compiler flags and link with --gc-sections flags that would eliminate all unneeded code and data (this would not be effective if all symbols were exported like now). Also to that end, we would also need to modify the linker script and add KEEP directive in relevant places to force the linker to keep certain code (for example interrupt or other exception handlers) from garbage collecting. For details see https://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf.

One disadvantage of hiding most symbols is that that the backtrace printed to the console when OSV or app on it crashed would be less friendly possible missing most symbols names like in this example:

OSv v0.55.0-15-g362accda
page fault outside application, addr: 0x0000000000000000
[registers]
RIP: 0x00000000403c64de <???+1077699806>
RFL: 0x0000000000010286  CS:  0x0000000000000008  SS:  0x0000000000000010
RAX: 0x0000004000000000  RBX: 0x0000000000020000  RCX: 0x0000000000000026  RDX: 0x0000000000000000
RSI: 0x0000000040687330  RDI: 0xffff800000013040  RBP: 0xffff80000010ffc0  R8:  0xffff800000165f90
R9:  0xffff800000016500  R10: 0x8000000000000000  R11: 0xffff800000084170  R12: 0xffff80000010ff90
Out of memory: could not reclaim any further. Current memory: -88 Kb
[backtrace]
0x00000000403c23c0 <???+1077683136>
0x00000000403c3d4f <???+1077689679>
0x00000000403c3e1f <???+1077689887>
0x00000000403dae6b <???+1077784171>
0x000000004037ea72 <???+1077406322>

vs

OSv v0.55.0-15-g67fcc08e
page fault outside application, addr: 0x0000000000000000
[registers]
RIP: 0x00000000403f951e <memory::page_pool::l2::refill()+206>
RFL: 0x0000000000010286  CS:  0x0000000000000008  SS:  0x0000000000000010
RAX: 0x0000004000000000  RBX: 0x0000000000020000  RCX: 0x0000000000000026  RDX: 0x0000000000000000
RSI: 0x0000000040912370  RDI: 0xffff800000013040  RBP: 0xffff80000010ffc0  R8:  0xffff80000007cf90
R9:  0xffff800000016500  R10: 0xOut of memory: could not reclaim any further. Current memory: -8 Kb
[backtrace]
0x00000000403f6620 <memory::oom()+32>
0x00000000403f779f <memory::reclaimer::_do_reclaim()+287>
0x00000000403f786f <???+1077901423>
0x000000004040ee3b <thread_main_c+43>
0x00000000403add32 <???+1077599538>

We could provide a script (tool) that could easily recreate more readable backtrace:

gdb -se=./build/release/loader.elf -ex 'info symbol 0x00000000403c23c0' -batch
memory::oom() + 32 in section .text

In most cases, the backtrace would still have at least the libc symbol name because that would still be present in .dynstr.