release builds no longer reproducible

[cyphar]

Our release binary builds are no longer reproducible, I expect this is either because libseccomp itself is not reproducible (possible but seems unlikely to me) or because of the new build process we've added some randomness (in the form of mktemp being used as the root for the libseccomp installation).

[kolyshkin]

I tried to get rid of mktemp:

diff --git a/script/release.sh b/script/release.sh
index b04fe009..5a118fac 100755
--- a/script/release.sh
+++ b/script/release.sh
@@ -33,7 +33,7 @@ function build_project() {
        local libseccomp_ver='2.5.1'
        local tarball="libseccomp-${libseccomp_ver}.tar.gz"
        local prefix
-       prefix="$(mktemp -d)"
+       prefix="$(pwd)/installed-libseccomp-${libseccomp_ver}"
        wget "https://github.com/seccomp/libseccomp/releases/download/v${libseccomp_ver}/${tarball}"{,.asc}
        tar xf "$tarball"
        (

but the runc binary created is still different every time, even the .note.go.buildid is different.

It looks like libseccomp.a is different each time:

[kir@kir-rhat runc]$ diff -u installed-libseccomp-2.5.1-sums-*
--- installed-libseccomp-2.5.1-sums-1	2021-05-20 11:35:25.668557703 -0700
+++ installed-libseccomp-2.5.1-sums-2	2021-05-20 11:36:13.283123263 -0700
@@ -34,6 +34,6 @@
 ce5189b7a2020f975d7595f08fc23fb9  ./share/man/man3/seccomp_api_get.3
 b73f883afdf3257ccf4bf574c4a4a850  ./share/man/man3/seccomp_attr_get.3
 e1d86b02f6d7aa6a144a435202cb047b  ./share/man/man3/seccomp_release.3
-a725359b9013ad315ac85c0c119042c1  ./lib/libseccomp.a
+af72aa8797abd70e706e726507e6cd10  ./lib/libseccomp.a
 4dc5426585cd92478df6af046955fe5a  ./lib/libseccomp.la
 dac487ad38831294625783c8b3cad9d3  ./lib/pkgconfig/libseccomp.pc

Looking into it, it seems that ar timestamps differ. I am not aware whether this can result in different go buildid.

Note that ar has D option for 0 timestamps and uids/gids, and this option can be enabled by default (when building binutils) but it seems on my Fedora 34 it is not.

This is where I stop.

[cyphar]

Yeah it seems that libseccomp doesn't build reproducibly. That's a bit of a shame. I'll open an upstream issue about it.

[tianon]

FWIW, Debian appears to be building 2.5.1, 2.3.3, and 2.3.1 reproducibly in different release pockets:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/libseccomp.html

Might be worth digging in https://salsa.debian.org/debian/libseccomp/-/tree/debian/sid/debian to see if there's anything special they do; although from my reading, it might even just be setting SOURCE_DATE_EPOCH appropriately? That debian/rules file looks pretty vanilla so it's bound to be something dpkg or debhelper do by default and SOURCE_DATE_EPOCH feels like the most likely candidate.

[kolyshkin]

I took another look at it. Managed to make libseccomp.a reproducible, and figured that a previous fix (#2947 (comment)) also helps.

I have also added -buildid= and -B=0x00 to -ldflags which seems to help, too.

So, 4 changes in total. But that is not enough :(

Here's my current patch:

diff --git a/script/release.sh b/script/release.sh
index b04fe009..ba0d9137 100755
--- a/script/release.sh
+++ b/script/release.sh
@@ -33,20 +33,22 @@ function build_project() {
        local libseccomp_ver='2.5.1'
        local tarball="libseccomp-${libseccomp_ver}.tar.gz"
        local prefix
-       prefix="$(mktemp -d)"
+       prefix="$(pwd)/installed-libseccomp-${libseccomp_ver}"
        wget "https://github.com/seccomp/libseccomp/releases/download/v${libseccomp_ver}/${tarball}"{,.asc}
        tar xf "$tarball"
        (
                cd "libseccomp-${libseccomp_ver}"
                ./configure --prefix="$prefix" --enable-static --disable-shared
                make install
+               # Strip ar dates.
+               objcopy --enable-deterministic-archives $prefix/lib/libseccomp.a
        )
        mv "$tarball"{,.asc} "$builddir"
 
        # Add -a to go build flags to make sure it links against
        # the provided libseccomp, not the system one (otherwise
        # it can reuse cached pkg-config results).
-       make -C "$root" PKG_CONFIG_PATH="${prefix}/lib/pkgconfig" COMMIT_NO= EXTRA_FLAGS="-a" static
+       make -C "$root" PKG_CONFIG_PATH="${prefix}/lib/pkgconfig" COMMIT_NO= EXTRA_FLAGS="-a" EXTRA_LDFLAGS="-buildid='' -B=0x00" static
        rm -rf "$prefix"
        mv "$root/$project" "$1"
 }

Let's see what's left.

[kir@kir-rhat runc]$ make clean
...
[kir@kir-rhat runc]$ ./script/release.sh 
...
[kir@kir-rhat runc]$ mv release 1-release
[kir@kir-rhat runc]$ make clean
...
[kir@kir-rhat runc]$ ./script/release.sh 
...
[kir@kir-rhat runc]$ diffoscope {,1-}release/1.0.0/runc.amd64
 |#                                                                                                                                                                 |  N/A%  None  ETA:  --:--:-- --- release/1.0.0/runc.amd64
+++ 1-release/1.0.0/runc.amd64
├── readelf --wide --debug-dump=rawline {}
│ @@ -6363,15 +6363,15 @@
│    Opcode 9 has 1 arg
│    Opcode 10 has 0 args
│    Opcode 11 has 0 args
│    Opcode 12 has 1 arg
│  
│   The Directory Table (offset 0x2656, lines 4, columns 1):
│    Entry	Name
│ -  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│    1	(indirect line string, offset: 0x455): /_/github.com/seccomp
│    2	(indirect line string, offset: 0x46b): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/lib/gcc/x86_64-redhat-linux/11/include
│    3	(indirect line string, offset: 0x4c5): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include
│  
│   The File Name Table (offset 0x266c, lines 4, columns 2):
│    Entry	Dir	Name
│    0	0	(indirect line string, offset: 0x500): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1_cgo_export.c
│ @@ -6458,15 +6458,15 @@
│    Opcode 9 has 1 arg
│    Opcode 10 has 0 args
│    Opcode 11 has 0 args
│    Opcode 12 has 1 arg
│  
│   The Directory Table (offset 0x26f7, lines 5, columns 1):
│    Entry	Name
│ -  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│    1	(indirect line string, offset: 0x455): /_/github.com/seccomp
│    2	(indirect line string, offset: 0x56a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include/bits
│    3	(indirect line string, offset: 0x5aa): /_/github.com/seccomp/libseccomp-golang@v0.9.1/home/kir/go/src/github.com/opencontainers/runc/installed-libseccomp-2.5.1/include
│    4	(indirect line string, offset: 0x4c5): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include
│  
│   The File Name Table (offset 0x2711, lines 6, columns 2):
│    Entry	Dir	Name
│ @@ -7285,15 +7285,15 @@
│    Opcode 9 has 1 arg
│    Opcode 10 has 0 args
│    Opcode 11 has 0 args
│    Opcode 12 has 1 arg
│  
│   The Directory Table (offset 0x2b6f, lines 7, columns 1):
│    Entry	Name
│ -  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +  0	(indirect line string, offset: 0x3dc): /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│    1	(indirect line string, offset: 0x6c7): /_/github.com/seccomp/libseccomp-golang@v0.9.1/home/kir/go/src/github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang
│    2	(indirect line string, offset: 0x455): /_/github.com/seccomp
│    3	(indirect line string, offset: 0x46b): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/lib/gcc/x86_64-redhat-linux/11/include
│    4	(indirect line string, offset: 0x56a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include/bits
│    5	(indirect line string, offset: 0x5aa): /_/github.com/seccomp/libseccomp-golang@v0.9.1/home/kir/go/src/github.com/opencontainers/runc/installed-libseccomp-2.5.1/include
│    6	(indirect line string, offset: 0x4c5): /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include
├── readelf --wide --debug-dump=info {}
│ @@ -14699,15 +14699,15 @@
│     Unit Type:     DW_UT_compile (1)
│     Abbrev Offset: 0x1c14
│     Pointer Size:  8
│   <0><8557>: Abbrev Number: 5 (DW_TAG_compile_unit)
│      <8558>   DW_AT_producer    : (indirect string, offset: 0x79): GNU C17 11.1.1 20210531 (Red Hat 11.1.1-3)
│      <855c>   DW_AT_language    : 29	(C11)
│      <855d>   DW_AT_name        : (indirect line string, offset: 0x52e): /_/github.com/seccomp/libseccomp-golang@v0.9.1_cgo_export.c
│ -    <8561>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +    <8561>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│      <8565>   DW_AT_low_pc      : 0x863a70
│      <856d>   DW_AT_high_pc     : 0x35
│      <8575>   DW_AT_stmt_list   : 0x2634
│   <1><8579>: Abbrev Number: 6 (DW_TAG_typedef)
│      <857a>   DW_AT_name        : (indirect string, offset: 0x497): size_t
│      <857e>   DW_AT_decl_file   : 2
│      <857f>   DW_AT_decl_line   : 209
│ @@ -14875,15 +14875,15 @@
│     Unit Type:     DW_UT_compile (1)
│     Abbrev Offset: 0x1cfd
│     Pointer Size:  8
│   <0><86c1>: Abbrev Number: 24 (DW_TAG_compile_unit)
│      <86c2>   DW_AT_producer    : (indirect string, offset: 0x79): GNU C17 11.1.1 20210531 (Red Hat 11.1.1-3)
│      <86c6>   DW_AT_language    : 29	(C11)
│      <86c7>   DW_AT_name        : (indirect line string, offset: 0x659): /_/github.com/seccomp/libseccomp-golang@v0.9.1seccomp.cgo2.c
│ -    <86cb>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +    <86cb>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│      <86cf>   DW_AT_low_pc      : 0x863ab0
│      <86d7>   DW_AT_high_pc     : 0x2ce
│      <86df>   DW_AT_stmt_list   : 0x26d5
│   <1><86e3>: Abbrev Number: 13 (DW_TAG_base_type)
│      <86e4>   DW_AT_byte_size   : 8
│      <86e5>   DW_AT_encoding    : 5	(signed)
│      <86e6>   DW_AT_name        : (indirect string, offset: 0x5): long int
│ @@ -16636,15 +16636,15 @@
│     Unit Type:     DW_UT_compile (1)
│     Abbrev Offset: 0x1ecd
│     Pointer Size:  8
│   <0><9537>: Abbrev Number: 22 (DW_TAG_compile_unit)
│      <9538>   DW_AT_producer    : (indirect string, offset: 0x79): GNU C17 11.1.1 20210531 (Red Hat 11.1.1-3)
│      <953c>   DW_AT_language    : 29	(C11)
│      <953d>   DW_AT_name        : (indirect line string, offset: 0x77f): /_/github.com/seccomp/libseccomp-golang@v0.9.1seccomp_internal.cgo2.c
│ -    <9541>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +    <9541>   DW_AT_comp_dir    : (indirect line string, offset: 0x40a): /_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│      <9545>   DW_AT_low_pc      : 0x863d80
│      <954d>   DW_AT_high_pc     : 0x2a7
│      <9555>   DW_AT_stmt_list   : 0x2b4d
│   <1><9559>: Abbrev Number: 10 (DW_TAG_base_type)
│      <955a>   DW_AT_byte_size   : 8
│      <955b>   DW_AT_encoding    : 5	(signed)
│      <955c>   DW_AT_name        : (indirect string, offset: 0x5): long int
├── strings --all --bytes=8 {}
│ @@ -71474,15 +71474,14 @@
│  t3j,M35*5Ti
│  :ZZ{I9U"
│  .wahx/X9
│  aJ>Y?Hd^Z
│  w{ENgaZfm:z
│  pXUpX@:,"
│  XetYetYetYetYetYetYetYetYe
│ -T}m/9W2V
│  em S|MQ.is
│  bV+~wE$^R.+
│  ]ye~LC%'=`
│  iI8m,GSSn
│  	U34Jc)d[
│  ipf,yy\"
│  }*d/|Z!Cs{/
├── readelf --wide --decompress --string-dump=.zdebug_line_str {}
│ @@ -67,15 +67,15 @@
│    [   340]  gcc_traceback.c
│    [   350]  gcc_util.c
│    [   35b]  linux_syscall.c
│    [   36b]  gcc_amd64.S
│    [   377]  enosys_linux.cgo2.c
│    [   38b]  /_/github.com/opencontainers/runc@/libcontainer/seccomp/patchbpf
│    [   3cc]  enosys_linux.go
│ -  [   3dc]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
│ +  [   3dc]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179
│    [   455]  /_/github.com/seccomp
│    [   46b]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/lib/gcc/x86_64-redhat-linux/11/include
│    [   4c5]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include
│    [   500]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1_cgo_export.c
│    [   56a]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/usr/include/bits
│    [   5aa]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/home/kir/go/src/github.com/opencontainers/runc/installed-libseccomp-2.5.1/include
│    [   62b]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1seccomp.cgo2.c

So, this looks like some stupid tmp dir which go creates when building libseccomp-golang which needs to be ironed out, and I am out of ideas for now.

release builds no longer reproducible

Were they?

[cyphar]

I'm quite confused why we need to add so many things to Go -- by default all Go binaries are reproducible? (Though maybe that guarantee doesn't exist for cgo-linked binaries?)

release builds no longer reproducible

Were they?

In the strictest meaning of the word, no. But previously if I ran script/release.sh twice, I'm fairly sure the binaries would have the same hash (though maybe I'm misremembering -- umoci has CI checks for this but we never had CI checks for such a thing in runc) -- though this was because we used the system's libseccomp library (or in the case of openSUSE, the system-installed one that I built because openSUSE doesn't ship static versions of libseccomp).

In any case, fixing this would be quite nice (the release script has another issue with $libdir but that's something else we can fix).

[kolyshkin]

previously if I ran script/release.sh twice, I'm fairly sure the binaries would have the same hash

This was probably just a side effect of go caching, which is no longer since commit 06a9ea3. IOW, script/release.sh && go clean -cache && script/release.sh would not result in the same binary (haven't checked that though).

[kolyshkin]

-  [   3dc]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build4092077316/b179
+  [   3dc]  /_/github.com/seccomp/libseccomp-golang@v0.9.1/_/github.com/seccomp/libseccomp-golang@v0.9.1/tmp/go-build3937345316/b179

Note the doubling of /_/github.com/seccomp/libseccomp-golang@v0.9.1 path -- this is probably the reason why it is not removed by -trimpath. Smell like some bug in golang toolchain, or something in our build environment that leads to that.

So I retried with go1.17beta1 and it's the same thing :(

[kolyshkin]

OK, the difference (#2947 (comment)) is debug info, so the stripped binary is the same!

[kolyshkin]

OK, here: #3099