Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: mkrpm.sh improvements #6196

Merged
merged 4 commits into from
Feb 8, 2024
Merged

Conversation

kmk3
Copy link
Collaborator

@kmk3 kmk3 commented Feb 6, 2024

This is a follow-up to #6126.

Relates to #297.

@kmk3 kmk3 requested a review from rusty-snake February 6, 2024 06:02
@kmk3
Copy link
Collaborator Author

kmk3 commented Feb 6, 2024

@rusty-snake By the way, do you use mkrpm.sh?

Copy link
Collaborator

@glitsj16 glitsj16 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I'm not very familiar with rpm-based Linuxes, this all looks pretty straightforward. LGTM

@@ -47,7 +47,7 @@ build_redhat_package:
- dnf update -y
- dnf install -y rpm-build gcc make
- ./ci/printenv.sh
- ./configure --prefix=/usr || (cat config.log; exit 1)
- ./configure || (cat config.log; exit 1)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this will install profiles in /usr/local/etc IIRC.

@rusty-snake
Copy link
Collaborator

By the way, do you use mkrpm.sh?

My own highly customized one.

@kmk3
Copy link
Collaborator Author

kmk3 commented Feb 7, 2024

@rusty-snake on Feb 6:

Note that this will install profiles in /usr/local/etc IIRC.

To clarify, whenever running ./configure -> make dist -> mkdeb.sh/mkrpm.sh
(as is done in .gitlab-ci.yml), the ./configure step is mostly just used to
generate config.mk (which in general needs to exist before calling make).

Passing --prefix=/usr to ./configure before calling mkdeb.sh/mkrpm.sh
doesn't do anything, as mkdeb.sh/mkrpm.sh always calls ./configure again (and
therefore overwrites config.mk if it exists, which is where ${prefix} is
defined).

I removed --quiet from rpmbuild now, so the above can be seen in the build
log.

Basically the workflow is:

  • ./configure: Generate config.mk
  • make dist: Include config.mk and generate the dist archive
  • mkdeb.sh/mkrpm.sh: Take the dist archive, extract it and run ./configure
    with distribution-specific arguments, make and make install

(Though currently mkrpm.sh generates a separate dist archive by itself)

From the build log of build_redhat_package:

$ ./configure || (cat config.log; exit 1)
[...]
checking for linux/seccomp.h... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating config.sh

Compile options:
   CC: gcc
   CFLAGS: -g -O2
   CPPFLAGS: 
   LDFLAGS: 
   EXTRA_CFLAGS:  -mindirect-branch=thunk -fstack-clash-protection -fstack-protector-strong
   DEPS_CFLAGS:  -MMD -MP
   EXTRA_LDFLAGS: 
   LIBS: 
   fatal warnings: 
   gcov instrumentation: 
   install as a SUID executable: -DHAVE_SUID
   install contrib scripts: yes
   prefix: /usr/local
   sysconfdir: ${prefix}/etc
   Spectre compiler patch: yes
$ make rpms
./platform/rpm/mkrpm.sh
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.MyB3M4
[...]
+ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --prefix=/usr --disable-userns --disable-contrib-install
configure: WARNING: unrecognized options: --disable-dependency-tracking
[...]
checking for linux/seccomp.h... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating config.sh
configure: WARNING: unrecognized options: --disable-dependency-tracking

Compile options:
   CC: gcc
   CFLAGS: -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
   CPPFLAGS: 
   LDFLAGS: -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 
   EXTRA_CFLAGS:  -fstack-clash-protection -fstack-protector-strong
   DEPS_CFLAGS:  -MMD -MP
   EXTRA_LDFLAGS: 
   LIBS: 
   fatal warnings: 
   gcov instrumentation: 
   install as a SUID executable: -DHAVE_SUID
   install contrib scripts: no
   prefix: /usr
   sysconfdir: /etc
   Spectre compiler patch: yes
[...]
+ make -j2
make[1]: Entering directory '/tmp/tmp.9aifQ1KP8u/BUILD/firejail-0.9.73'
[...]
+ /usr/bin/make install DESTDIR=/tmp/tmp.9aifQ1KP8u/BUILDROOT/firejail-0.9.73-1.x86_64 'INSTALL=/usr/bin/install -p'
make[1]: Entering directory '/tmp/tmp.9aifQ1KP8u/BUILD/firejail-0.9.73'

Though from the output above it seems that rpmbuild itself already passes
--prefix=/usr to ./configure:

+ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --prefix=/usr --disable-userns --disable-contrib-install
[...]
   prefix: /usr

In any case, I'll post a PR after this one that makes CI always pass
--prefix=/usr --enable-fatal-warnings to mkdeb.sh/mkrpm.sh for consistency
and to make this clearer.

To make the CI logs more informative, as currently nothing from the
build itself is shown.

Added on commit d684d99 ("Fix mkrpm.sh", 2016-02-16) / PR netblue30#297.
To abort the build if any error occurs.

See also commit 7d9db83 ("fail build if any step in the script fails",
2019-06-21).
They are not being properly forwarded to mkrpm.sh (which re-runs
./configure before the actual build), so just remove them for now.
@rusty-snake
Copy link
Collaborator

Though from the output above it seems that rpmbuild itself already passes
--prefix=/usr to ./configure:

If %configure is used yes.

@kmk3
Copy link
Collaborator Author

kmk3 commented Feb 8, 2024

Though from the output above it seems that rpmbuild itself already passes
--prefix=/usr to ./configure:

If %configure is used yes.

I see; removed the commit that adds another --prefix=/usr, to avoid passing
the same argument twice.

@kmk3 kmk3 merged commit 3b89a75 into netblue30:master Feb 8, 2024
2 checks passed
@kmk3 kmk3 deleted the build-mkrpm-improvements branch February 8, 2024 06:24
kmk3 added a commit that referenced this pull request Feb 8, 2024
@rusty-snake
Copy link
Collaborator

@kmk3 FYI rpm -E %configure on Fedora 39:

  CFLAGS="${CFLAGS:--O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer }" ; export CFLAGS ; 
  CXXFLAGS="${CXXFLAGS:--O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer }" ; export CXXFLAGS ; 
  FFLAGS="${FFLAGS:--O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules }" ; export FFLAGS ; 
  FCFLAGS="${FCFLAGS:--O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules }" ; export FCFLAGS ; 
  VALAFLAGS="${VALAFLAGS:--g}" ; export VALAFLAGS ; 
  RUSTFLAGS="${RUSTFLAGS:--Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-Wl,-z,relro -Clink-arg=-Wl,-z,now --cap-lints=warn}" ; export RUSTFLAGS ; 
  LDFLAGS="${LDFLAGS:--Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1  }" ; export LDFLAGS ; 
  LT_SYS_LIBRARY_PATH="${LT_SYS_LIBRARY_PATH:-/usr/lib64:}" ; export LT_SYS_LIBRARY_PATH ; 
  CC="${CC:-gcc}" ; export CC ; 
  CXX="${CXX:-g++}" ; export CXX; 
  [ "-flto=auto -ffat-lto-objects"x != x ] && 
  for file in $(find . -type f -name configure -print); do 
    /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\) = /__attribute__ ((used)) char (*f) () = /g' $file; 
    diff -u $file.backup $file && mv $file.backup $file 
    /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\);/__attribute__ ((used)) char (*f) ();/g' $file; 
    diff -u $file.backup $file && mv $file.backup $file 
    /usr/bin/sed -r --in-place=.backup 's/^char \$2 \(\);/__attribute__ ((used)) char \$2 ();/g' $file; 
    diff -u $file.backup $file && mv $file.backup $file 
    /usr/bin/sed --in-place=.backup '1{$!N;$!N};$!N;s/int x = 1;\nint y = 0;\nint z;\nint nan;/volatile int x = 1; volatile int y = 0; volatile int z, nan;/;P;D' $file; 
    diff -u $file.backup $file && mv $file.backup $file 
    /usr/bin/sed --in-place=.backup 's#^lt_cv_sys_global_symbol_to_cdecl=.*#lt_cv_sys_global_symbol_to_cdecl="sed -n -e '"'"'s/^T .* \\(.*\\)$/extern int \\1();/p'"'"' -e '"'"'s/^$symcode* .* \\(.*\\)$/extern char \\1;/p'"'"'"#' $file; 
    diff -u $file.backup $file && mv $file.backup $file 
  done; 
  [ "1" = 1 ] && for i in $(find $(dirname ./configure) -name config.guess -o -name config.sub) ; do 
      [ -f /usr/lib/rpm/redhat/$(basename $i) ] && /usr/bin/rm -f $i && /usr/bin/cp -fv /usr/lib/rpm/redhat/$(basename $i) $i ; 
  done ; 
  [ "1" = 1 ] && [ x != "x-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld" ] && 
      for i in $(find . -name ltmain.sh) ; do 
        /usr/bin/sed -i.backup -e 's~compiler_flags=$~compiler_flags="-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld"~' $i 
      done ; 
  ./configure --build=x86_64-redhat-linux --host=x86_64-redhat-linux \
	--program-prefix= \
	--disable-dependency-tracking \
	 \
	--prefix=/usr \
	--exec-prefix=/usr \
	--bindir=/usr/bin \
	--sbindir=/usr/sbin \
	--sysconfdir=/etc \
	--datadir=/usr/share \
	--includedir=/usr/include \
	--libdir=/usr/lib64 \
	--libexecdir=/usr/libexec \
	--localstatedir=/var \
	$(grep -q "runstatedir=DIR" ./configure && echo '--runstatedir=/run') \
	--sharedstatedir=/var/lib \
	--mandir=/usr/share/man \
	--infodir=/usr/share/info

@kmk3
Copy link
Collaborator Author

kmk3 commented Feb 11, 2024

@rusty-snake on Feb 9:

FYI rpm -E %configure on Fedora 39:

Thanks for the details.

  CFLAGS="${CFLAGS:--O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer }" ; export CFLAGS ; 

Good to know the default flags. The hardening seems similar to what is already
done.

Maybe -fcf-protection could also be added to the default build?

  ./configure --build=x86_64-redhat-linux --host=x86_64-redhat-linux \
	--program-prefix= \
	--disable-dependency-tracking \
	 \
	--prefix=/usr \
	--exec-prefix=/usr \
	--bindir=/usr/bin \
	--sbindir=/usr/sbin \
	--sysconfdir=/etc \
	--datadir=/usr/share \
	--includedir=/usr/include \
	--libdir=/usr/lib64 \
	--libexecdir=/usr/libexec \
	--localstatedir=/var \
	$(grep -q "runstatedir=DIR" ./configure && echo '--runstatedir=/run') \
	--sharedstatedir=/var/lib \
	--mandir=/usr/share/man \
	--infodir=/usr/share/info

Strangely --runstatedir=/run does not appear in the log for
build_redhat_package:

+ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --prefix=/usr --disable-userns --disable-contrib-install
[...]
   prefix: /usr

Even though it is defined in the committed script:

$ git grep -n "runstatedir=DIR" ./configure
configure:1362:  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]

This might be an issue if the directory is eventually made configurable (such
as for #4535).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done (on RELNOTES)
Development

Successfully merging this pull request may close these issues.

3 participants