Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nginx:1.25.5 fails to build on arm32v5 #881

Closed
yosifkit opened this issue Apr 19, 2024 · 9 comments
Closed

nginx:1.25.5 fails to build on arm32v5 #881

yosifkit opened this issue Apr 19, 2024 · 9 comments

Comments

@yosifkit
Copy link
Contributor

Describe the bug

Building the mainline/debian/Dockerfile (at nginx 1.25.5) causes an error in linking with undefined references. It seems like a missing dependency.

Log:

Also available on the DOI jenkins instance (at least for a little while) https://doi-janky.infosiftr.net/job/wip/job/new/job/build-arm32v5/1693/.

make[1]: *** [Makefile:272: run-test262] Error 1
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: /tmp/cc3YvmNt.ltrans12.ltrans.o: in function `js_atomics_op.lto_priv.0':
/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55162: undefined reference to `__atomic_load_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55148: undefined reference to `__atomic_fetch_xor_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55149: undefined reference to `__atomic_exchange_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55189: undefined reference to `__atomic_compare_exchange_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55144: undefined reference to `__atomic_fetch_add_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55145: undefined reference to `__atomic_fetch_and_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55146: undefined reference to `__atomic_fetch_or_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55147: undefined reference to `__atomic_fetch_sub_8'
/usr/bin/ld: /tmp/cc3YvmNt.ltrans12.ltrans.o: in function `js_atomics_store.lto_priv.0':
/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55253: undefined reference to `__atomic_store_8'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:214: qjsc] Error 1
make[1]: Leaving directory '/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13'
make: *** [debian/rules:48: config.pre.nginx] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Build command 'cd nginx-module-njs-1.25.5+0.8.4 && dpkg-buildpackage -b -uc' failed.

To reproduce

Steps to reproduce the behavior:

(either on a native host or one with qemu user mode emulation)

  1. docker build --platform linux/arm/v5 --progress=plain mainline/debian
  2. wait ... wait some more ...
  3. See error

Expected behavior

A built arm32v5 image at the end.

Your environment

  • docker 26.0.0 on Docker Desktop (Windows) and on the DOI builder running Docker 20.10
  • nginx:1.25.5
  • platform: arm32v5
@oxpa
Copy link
Collaborator

oxpa commented Apr 20, 2024

Thank you for the report.

I saw the build problem (it failed a build of rootless nginx container) but didn't have a chance to look into this.
Will do my best to fix it next week.

@thresheek
Copy link
Collaborator

Thanks for the report @yosifkit - much appreciated. We do indeed not test our builds on armv5...

@thresheek
Copy link
Collaborator

Hopefully fixed by 3fb7e2e and https://hg.nginx.org/pkg-oss/rev/93ac6e194ad0, let's wait for actual builds to commence though

@thresheek thresheek mentioned this issue Apr 23, 2024
Merged
@thresheek
Copy link
Collaborator

hmm, it seems the fix wasnt enough, at least it's still failing (differently though):

ar -r -c build/libnjs.a \
	build/src/njs_diyfp.o build/src/njs_dtoa.o build/src/njs_dtoa_fixed.o build/src/njs_str.o build/src/njs_strtod.o build/src/njs_murmur_hash.o build/src/njs_djb_hash.o build/src/njs_utf8.o build/src/njs_utf16.o build/src/njs_arr.o build/src/njs_rbtree.o build/src/njs_flathsh.o build/src/njs_trace.o build/src/njs_random.o build/src/njs_malloc.o build/src/njs_mp.o build/src/njs_sprintf.o build/src/njs_utils.o build/src/njs_chb.o build/src/njs_value.o build/src/njs_vm.o build/src/njs_vmcode.o build/src/njs_lexer.o build/src/njs_lexer_keyword.o build/src/njs_parser.o build/src/njs_variable.o build/src/njs_scope.o build/src/njs_generator.o build/src/njs_disassembler.o build/src/njs_module.o build/src/njs_extern.o build/src/njs_boolean.o build/src/njs_number.o build/src/njs_symbol.o build/src/njs_string.o build/src/njs_object.o build/src/njs_object_prop.o build/src/njs_array.o build/src/njs_json.o build/src/njs_function.o build/src/njs_regexp.o build/src/njs_date.o build/src/njs_error.o build/src/njs_math.o build/src/njs_array_buffer.o build/src/njs_typed_array.o build/src/njs_promise.o build/src/njs_encoding.o build/src/njs_iterator.o build/src/njs_async.o build/src/njs_builtin.o build/external/njs_regex.o build/src/njs_buffer.o build/external/njs_crypto_module.o build/external/njs_md5.o build/external/njs_sha1.o build/external/njs_sha2.o build/external/njs_webcrypto_module.o build/external/njs_xml_module.o build/external/njs_zlib_module.o build/external/njs_fs_module.o build/external/njs_query_string_module.o build/build/njs_modules.o 
cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/lib/quickjs/ -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 \
	-pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC \
	external/njs_shell.c \
	build/libnjs.a \
	-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/lib/quickjs/ -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2  -lz  -ledit
lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
/usr/bin/ld: /tmp/ccw77qKL.ltrans11.ltrans.o: undefined reference to symbol '__atomic_fetch_sub_8@@LIBATOMIC_1.0'
/usr/bin/ld: /lib/arm-linux-gnueabi/libatomic.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [build/Makefile:792: build/njs] Error 1
make[1]: Leaving directory '/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/extra/njs-0.8.4'
make: *** [debian/rules:48: config.pre.nginx] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Build command 'cd nginx-module-njs-1.26.0+0.8.4 && dpkg-buildpackage -b -uc' failed.

Seems like debian's patch to quickjs also adds -latomic to HOST_LIBS: https://salsa.debian.org/a11y-team/quickjs/-/commit/7b8e2c8aa73f4c3196cd7f340e1779837bf9e32c

@thresheek
Copy link
Collaborator

Reproduced locally, it seems like we'd need to provide -latomic for njs linking step as well..

@thresheek
Copy link
Collaborator

And of course simply adding -latomic to njs' --ld-opt is not enough, since those options will be added before extra-linked libraries, so it will not be picked up by ld:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# hg diff
diff -r 93ac6e194ad0 debian/Makefile.module-njs
--- a/debian/Makefile.module-njs        Tue Apr 23 18:58:26 2024 +0100
+++ b/debian/Makefile.module-njs        Wed Apr 24 19:04:17 2024 +0000
@@ -32,7 +32,7 @@
        cd $$(CURDIR)/debian/extra/njs-$(NJS_VERSION) \&\& \
        ./configure \
                --cc-opt="$$(CFLAGS) -I$$(BUILDDIR_nginx)/extra/prebuilt/include/quickjs/" \
-               --ld-opt="$$(LDFLAGS) -L$$(BUILDDIR_nginx)/extra/prebuilt/lib/quickjs/" \&\& \
+               --ld-opt="$$(LDFLAGS) -L$$(BUILDDIR_nginx)/extra/prebuilt/lib/quickjs/ -latomic" \&\& \
        make -j$$(NUMJOBS) njs \&\& \
        mv build build-cli
 endef

Resulting in:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# ar -r -c build/libnjs.a \
        build/src/njs_diyfp.o build/src/njs_dtoa.o build/src/njs_dtoa_fixed.o build/src/njs_str.o build/src/njs_strtod.o build/src/njs_murmur_hash.o build/src/njs_djb_hash.o build/src/njs_utf8.o build/src/njs_utf16.o build/src/njs_arr.o build/src/njs_rbtree.o build/src/njs_flathsh.o build/src/njs_trace.o build/src/njs_random.o build/src/njs_malloc.o build/src/njs_mp.o build/src/njs_sprintf.o build/src/njs_utils.o build/src/njs_chb.o build/src/njs_value.o build/src/njs_vm.o build/src/njs_vmcode.o build/src/njs_lexer.o build/src/njs_lexer_keyword.o build/src/njs_parser.o build/src/njs_variable.o build/src/njs_scope.o build/src/njs_generator.o build/src/njs_disassembler.o build/src/njs_module.o build/src/njs_extern.o build/src/njs_boolean.o build/src/njs_number.o build/src/njs_symbol.o build/src/njs_string.o build/src/njs_object.o build/src/njs_object_prop.o build/src/njs_array.o build/src/njs_json.o build/src/njs_function.o build/src/njs_regexp.o build/src/njs_date.o build/src/njs_error.o build/src/njs_math.o build/src/njs_array_buffer.o build/src/njs_typed_array.o build/src/njs_promise.o build/src/njs_encoding.o build/src/njs_iterator.o build/src/njs_async.o build/src/njs_builtin.o build/external/njs_regex.o build/src/njs_buffer.o build/external/njs_crypto_module.o build/external/njs_md5.o build/external/njs_sha1.o build/external/njs_sha2.o build/external/njs_webcrypto_module.o build/external/njs_xml_module.o build/external/njs_zlib_module.o build/external/njs_fs_module.o build/external/njs_query_string_module.o build/build/njs_modules.o
cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 -I/usr/include/libxml2 \
        -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC \
        external/njs_shell.c \
        build/libnjs.a \
        -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2 -lz -ledit


lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
/usr/bin/ld: /tmp/ccoES5lX.ltrans11.ltrans.o: undefined reference to symbol '__atomic_fetch_sub_8@@LIBATOMIC_1.0'
/usr/bin/ld: /usr/lib/gcc/arm-linux-gnueabi/12/libatomic.so: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

And if we move -latomic to the end of libs linked, it's all good:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 -I/usr/include/libxml2      -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC  external/njs_shell.c    build/libnjs.a  -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2 -lz -ledit -latomic
lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4#

So that means we'd need a way to add it to EXTRA_LIBS in njs, which probably needs some extra patching.

@thresheek
Copy link
Collaborator

I just spoke to njs maintainer and I think we're going to add a configure check in njs to enable -latomic when linking on armv7l...

@thresheek
Copy link
Collaborator

So we discussed the approach on fixing it in upstream njs through configure checks and it felt a bit hacky to do so - especially keeping in mind that armel can be "multiarch" compiled on e.g. armv8 so multiple checks will be needed to fully cover the problem.

Long story short I've pushed a hack to our packaging which is a somewhat appropriate place to put hacks to :) https://hg.nginx.org/pkg-oss/rev/7bc1410f615c

I'll follow up with updating this repo and doing docker-library PR in a next few days.

@yosifkit
Copy link
Contributor Author

yosifkit commented May 6, 2024

Success! https://hub.docker.com/_/nginx/tags?page=&page_size=&ordering=&name=1.25.5

@yosifkit yosifkit closed this as completed May 6, 2024
@thresheek thresheek mentioned this issue Aug 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yosifkit @oxpa @thresheek