Allow running a statically-compiled Linux executable

[ERnsTL]

OSv currently can only run position-independent shared objects.

This means we usually can't run a random Linux application without recompiling it (with -fPIC -shared), unless we're lucky and the application is already mostly a .so (like is the case in the JVM).

It is, as detailed below, desireable to allow running a single statically compiled executable, as created for example by the Go programming language (Golang).

The Go programming language was created with the needs of server farms at Google, cloud computing, multicore processors and concurrency in mind and excels for use to write network servers and is positioned as such. Consequentially, its popularity has grown along with the trend in virtualization and cloud computing.

Running virtualized network servers in the cloud efficiently is also a primary use-case of OSv.

Because of this use-case overlap and bright future for the Go language in cloud computing, I created this ticket in the hope that the feature of running its statically compiled executables will be given an amount of priority and be available in one of the next releases.

This issue is possibly related to:

• Issue #190

[penberg]

👍 to this and to #190. No we just need to find someone to implement it!

[glommer]

That's actually one of the ideas for our GSoC application.

On Thu, Feb 13, 2014 at 5:48 PM, Pekka Enberg notifications@github.comwrote:

[image: 👍] to this and to #190#190.
No we just need to find someone to implement it!

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-34978995
.

[nyh]

Hi, can you please explain how this issue is different from #190? I see you
even copied a verbatim paragraph from it?

Is the difference the word static? Do you actually mean a static executable
where the libc calls have been replaced by code from glibc? I am afraid
doing this on osv is a big diversion from the way we are heading. Why is
it even necessary? What's wrong with dynamic executables, linked to
libc.so and not libc.a?

[nyh]

Now that I'm back to a computer with a decent keyboard, I can expand on my previous answer.

First, can you please explain to me why the Go people even want static linking? Static linking has so many problems: It generates huge executables, it waste tons of memory when different processes running together cannot share text pages ("shared libraries" got their name for a reason). It creates security problems when it's not enough to update a library, you also need to update all the programs that used it. Almost 10 years ago, Solaris 10 came out with an announcement: Static libc is no longer provided. You can no longer create static executables (if they use libc). Hardly anyone cried. Why would Go want to bring it back? Is it possible to compile go programs so that its runtime does not run Linux system calls directly, but rather use glibc (and links to it dynamically)? If I understand correctly, this is possible (with gccgo), so what's the benefit of not doing this?

Second, I want to explain what it would take to get OSv to run executables which are statically linked with libc.a - or executables which directly use Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc() function, we run OSv's code which does malloc(), which we wrote from scratch to fit the OSv design. If we statically compile with libc.a, we get some glibc's malloc() code which we can't control, which ultimatively calls the sbrk() system call. OSv does not currently support sbrk() (or even this concept), nor does it even support system calls in the Linux sense. We'll need to change all of this to run static executables. Looks really complex, and I wonder if actually worth it.

[penberg]

@nyh I assume it's to avoid Go version dependency issues but I couldn't really find a reference for it. Not that it really matters much why Go decided to go for static linking. If we want to run Go programs on OSv, that's what we have to support.

[glommer]

On Thu, Feb 13, 2014 at 11:11 PM, nyh notifications@github.com wrote:

Now that I'm back to a computer with a decent keyboard, I can expand on my
previous answer.

First, can you please explain to me why the Go people even want static
linking? Static linking has so many problems: It generates huge
executables, it waste tons of memory when different processes running
together cannot share text pages ("shared libraries" got their name for a
reason). It creates security problems when it's not enough to update a
library, you also need to update all the programs that used it. Almost 10
years ago, Solaris 10 came out with an announcement: Static libc is no
longer provided. You can no longer create static executables (if they use
libc). Hardly anyone cried. Why would Go want to bring it back? Is it
possible to compile go programs so that its runtime does not run Linux
system calls directly, but rather use glibc (and links to it dynamically)?
If I understand correctly, this is possible (with gccgo), so what's the
benefit of not doing this?

Nadav. wrt sharing pages: what are you talking about? For us that's
actually perfect. There is no one else in the system for you to share
with, so the distinction here is moot. That's a perfect combination
between go and OSv, in that aspect.

Second, I want to explain what it would take to get OSv to run executables
which are statically linked with libc.a - or executables which directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a, we get
some glibc's malloc() code which we can't control, which ultimatively calls
the sbrk() system call. OSv does not currently support sbrk() (or even this
concept), nor does it even support system calls in the Linux sense. We'll
need to change all of this to run static executables. Looks really complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of our
capabilities. What's different about sbrk besides it being rare?

[gleb-cloudius]

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run executables
which are statically linked with libc.a - or executables which directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a, we get
some glibc's malloc() code which we can't control, which ultimatively calls
the sbrk() system call. OSv does not currently support sbrk() (or even this
concept), nor does it even support system calls in the Linux sense. We'll
need to change all of this to run static executables. Looks really complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is
probably possible to force glibc to always use mmap by setting MALLOC_MMAP_THRESHOLD_
to a very small value, but does go uses glibc memory allocator at all?

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov notifications@github.comwrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a,
we get
some glibc's malloc() code which we can't control, which ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

[gleb-cloudius]

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov notifications@github.comwrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a,
we get
some glibc's malloc() code which we can't control, which ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still fits your
definition of rare I do not know.

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or
even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it.
But we
have made the commitment to support the Linux interface to the best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

This is what I mean by rare. The rest is a semantic discussion I have no
interest in carrying on.

Gleb.

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-35109330
.

[gleb-cloudius]

On Fri, Feb 14, 2014 at 10:16:56AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or
even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it.
But we
have made the commitment to support the Linux interface to the best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

That's exactly what Nadav was explaining is not the case for statically
linked executables, brk() was just an example and in reality should be
easy to implement. More complicated example may be clone() system call
which is used to implement pthread_create().

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:43 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:16:56AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov <notifications@github.com
wrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <
notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the
malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk()
(or
even
this
concept), nor does it even support system calls in the Linux
sense.
We'll
need to change all of this to run static executables. Looks
really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use
it.
But we
have made the commitment to support the Linux interface to the
best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

That's exactly what Nadav was explaining is not the case for statically
linked executables, brk() was just an example and in reality should be
easy to implement. More complicated example may be clone() system call
which is used to implement pthread_create().

ok, that's a bitch, indeed.

Gleb.

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-35112070
.

[gebi]

JFTR... ( @nyh ) golang executables, as produced by the native go toolchain, are not only statically compiled but do NOT use glibc at all. Everything is implemented without any libc call, yes syscalls too. If you use an up to date go toolchain you can also compile a go executable without using the libc resolver at all. Instead an internal one will be used, producing static executables even when network libraries are used.

IMHO the most elegant way to integrate golang with OSv would be to add another architecture to the runtime which than can use native OSv methods to access the environment (network, ...).
One architecture which may help could be the nacl architecture which got merged recently (compiling go executables to run inside the chrome native client sandbox).

[elazarl]

@nyh Reasoning behind Go's decision to use static executable is, having a very stable execution among different servers. If the kernel is identical (and mostly it is), there could be little differences between running it in one server or another. One of the JVM guys who now works on Go mentioned in a talk his frustration of finding subtle bugs when moving a JVM program between different JVMs among different servers.

You can see Rob Pike's objection to dynamic linking (back in plan9, and obviously now as one of Go's authors) here

[nyh]

On Thu, Jul 3, 2014 at 10:04 AM, Elazar Leibovich notifications@github.com
wrote:

You can see Rob Pike's objection to dynamic linking (back in plan9, and
obviously now as one of Go's authors) here
http://harmful.cat-v.org/software/dynamic-linking/

With all due respect to Rob Pike (and I have a lot of it!), I remember
using his Unix which didn't support shared libraries, and later moving to
Unix variants that did (SunOS, System Vr4, and finally Linux), and I really
welcomed this addition. Especially when it was first introduced (when every
megabyte counted), but I do think they are still valuable today - e.g., for
allowing you to fix a library without recompiling all the software on your
system.
But I guess this isn't the question now... If Go uses static executables,
and people want to run them on OSv, we need to do this :(

Nadav Har'El
nyh@cloudius-systems.com