3.9.0: tests make build non-reproducible

[dvzrv]

When building 3.9.0 as a package (using the test-data 3.0.0) for Arch Linux I ran our reproducible builds tools against the resulting package.
Beforehand I had to apply #2318 as install of the pkgconfig file fails otherwise.

What is the issue you have?

Running the tests (ctest --output-on-failure -LE git_required) after build renders the package unreproducible and alters the pkgconfig file to change cflags from -I/usr/include to -I/usr/local/include (amongst others).

Please describe the steps to reproduce the issue.

  cmake -DCMAKE_INSTALL_PREFIX='/usr' \
        -DCMAKE_INSTALL_LIBDIR='/usr/lib' \
        -DCMAKE_BUILD_TYPE='None' \
        -DBUILD_TESTING=ON \
        -DJSON_BuildTests=ON \
        -DJSON_MultipleHeaders=ON \
        -Wno-dev \
        -B build \
        -S .
  make VERBOSE=1 -C build
  ctest --output-on-failure -LE git_required
  make DESTDIR="${pkgdir}" install -C build

After a package has been created (building in a clean systemd-nspawn container), the package is rebuilt using the exact same dependencies with the repro tool.

Can you provide a small but working code example?

n/a

What is the expected behavior?

Package is reproducible. Tests do not alter the build and resulting installed files.

And what is the actual behavior instead?

This is the output of diffoscope on the two packages:

--- nlohmann-json-3.9.0-1-any.pkg.tar.zst
+++ build/nlohmann-json-3.9.0-1-any.pkg.tar.zst
├── nlohmann-json-3.9.0-1-any.pkg.tar
│ ├── file list
│ │ @@ -1,9 +1,9 @@
│ │  -rw-r--r--   0 root         (0) root         (0)     4659 2020-07-29 06:45:59.000000 .BUILDINFO
│ │ --rw-r--r--   0 root         (0) root         (0)     3854 2020-07-29 06:45:59.000000 .MTREE
│ │ +-rw-r--r--   0 root         (0) root         (0)     3858 2020-07-29 06:45:59.000000 .MTREE
│ │  -rw-r--r--   0 root         (0) root         (0)      337 2020-07-29 06:45:59.000000 .PKGINFO
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/nlohmann/
│ │  -rw-r--r--   0 root         (0) root         (0)     1504 2020-07-29 06:45:59.000000 usr/include/nlohmann/adl_serializer.hpp
│ │  -rw-r--r--   0 root         (0) root         (0)     4633 2020-07-29 06:45:59.000000 usr/include/nlohmann/byte_container_with_subtype.hpp
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/nlohmann/detail/
│ │ @@ -50,17 +50,17 @@
│ │  -rw-r--r--   0 root         (0) root         (0)    79525 2020-07-29 06:45:59.000000 usr/include/nlohmann/thirdparty/hedley/hedley.hpp
│ │  -rw-r--r--   0 root         (0) root         (0)     4981 2020-07-29 06:45:59.000000 usr/include/nlohmann/thirdparty/hedley/hedley_undef.hpp
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/cmake/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/
│ │  -rw-r--r--   0 root         (0) root         (0)      620 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake
│ │  -rw-r--r--   0 root         (0) root         (0)      675 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonConfigVersion.cmake
│ │ --rw-r--r--   0 root         (0) root         (0)     3599 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │ +-rw-r--r--   0 root         (0) root         (0)     3293 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/
│ │ --rw-r--r--   0 root         (0) root         (0)       97 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/nlohmann_json.pc
│ │ +-rw-r--r--   0 root         (0) root         (0)       91 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/nlohmann_json.pc
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/doc/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/
│ │  -rw-r--r--   0 root         (0) root         (0)     3213 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/CODE_OF_CONDUCT.md
│ │  -rw-r--r--   0 root         (0) root         (0)    92324 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/README.md
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/licenses/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/licenses/nlohmann-json/
│ ├── .MTREE
│ │ ├── .MTREE-content
│ │ │ @@ -1,11 +1,11 @@
│ │ │  #mtree
│ │ │  /set type=file uid=0 gid=0 mode=644
│ │ │  ./.BUILDINFO time=1596005159.0 size=4659 md5digest=63c650b55cdce35f19383793997e9f66 sha256digest=ddc54c9966f2bdd79f897309ed4fef322b885cfa94ba0b69e244e6c5c20cf8bc
│ │ │ -./.PKGINFO time=1596005159.0 size=337 md5digest=62f9e8e0245fd2e8ecebde2373d1bce5 sha256digest=051095f02018bb81a86be4e3baebd3089f975e329d1edf7fc5edc0ed3a48f5f0
│ │ │ +./.PKGINFO time=1596005159.0 size=337 md5digest=cbe4f20312aee609d4c4299824b7e7a0 sha256digest=619dae38f84bdc895d1a043e3247602896bdd37ff635affef83dd3f36f9746cd
│ │ │  /set mode=755
│ │ │  ./usr time=1596005159.0 type=dir
│ │ │  ./usr/include time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/include/nlohmann time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/include/nlohmann/adl_serializer.hpp time=1596005159.0 size=1504 md5digest=6681125504fe517fb4349916c7ca3ad5 sha256digest=9a09ae189930ebf891942ce0fb3b10674a0ae8d6e3c1db618d46c5357960c526
│ │ │  ./usr/include/nlohmann/byte_container_with_subtype.hpp time=1596005159.0 size=4633 md5digest=3773ad72c762eeada43f95ac0a0351ed sha256digest=9ceb0600c83ae0ec49701a6fc0aaa2f4e30f4dfd1120fb7d513f15b4c6c9cc37
│ │ │ @@ -55,17 +55,17 @@
│ │ │  /set mode=755
│ │ │  ./usr/lib time=1596005159.0 type=dir
│ │ │  ./usr/lib/cmake time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/lib/cmake/nlohmann_json time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake time=1596005159.0 size=620 md5digest=19b85ca4287a9d187a335feafbf45568 sha256digest=c7f377920b46cd6c89c6f2f1bccbd1cf7380addf545db0bf7046a27920e2bdd3
│ │ │  ./usr/lib/cmake/nlohmann_json/nlohmann_jsonConfigVersion.cmake time=1596005159.0 size=675 md5digest=bfc688b969c471dd5dcb38f6d9e823c7 sha256digest=ca77a7cb7c35cff0fd7e180b5f5a90955d0fa6a3b72909eccb4bcd62b36f449b
│ │ │ -./usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake time=1596005159.0 size=3599 md5digest=fc37818baa390ff80790e74713840ee7 sha256digest=427752e51feeefacb23ce487de7f888a552ea3181e0951600683814e68f676dd
│ │ │ +./usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake time=1596005159.0 size=3293 md5digest=a408eb50f0acc70cadd0e0cab904f0d5 sha256digest=cc206c5bd4cc8ec064d6174d7664377f8dcd5ed280476c4b7473154d0996bb11
│ │ │  ./usr/lib/pkgconfig time=1596005159.0 mode=755 type=dir
│ │ │ -./usr/lib/pkgconfig/nlohmann_json.pc time=1596005159.0 size=97 md5digest=8bdec2b26a228ff226cc21cf26937e84 sha256digest=a30c5a41abc1efef47ed270154cdeddf9790c838e219b164b03700b318c4690b
│ │ │ +./usr/lib/pkgconfig/nlohmann_json.pc time=1596005159.0 size=91 md5digest=8a8567f3f10d22c334c5804521a33788 sha256digest=eb5f758dfd576d338954329a2eb4e9b6f8518bf107126b8acae19b5c33388a8e
│ │ │  /set mode=755
│ │ │  ./usr/share time=1596005159.0 type=dir
│ │ │  ./usr/share/doc time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/share/doc/nlohmann-json time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/share/doc/nlohmann-json/CODE_OF_CONDUCT.md time=1596005159.0 size=3213 md5digest=0227ed5d32159cf5923d5b71e8c82c12 sha256digest=3f82f9935dbdd87919ec79bf0f2a43232f2cfe82040236900652458ffff77783
│ │ │  ./usr/share/doc/nlohmann-json/README.md time=1596005159.0 size=92324 md5digest=07df80351c6c3d2f1aeb8b5a0ef87f1d sha256digest=174c18ebde64965cac98c147204e1e2c33714f4d93242bf375a2923ea9a37f71
│ ├── .PKGINFO
│ │ @@ -3,12 +3,12 @@
│ │  pkgname = nlohmann-json
│ │  pkgbase = nlohmann-json
│ │  pkgver = 3.9.0-1
│ │  pkgdesc = JSON for Modern C++
│ │  url = https://github.com/nlohmann/json
│ │  builddate = 1596005159
│ │  packager = David Runge <dvzrv@archlinux.org>
│ │ -size = 1025469
│ │ +size = 1025157
│ │  arch = any
│ │  license = MIT
│ │  makedepend = cmake
│ │  makedepend = git
│ ├── usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │ @@ -37,22 +37,16 @@
│ │    message(FATAL_ERROR "Some (but not all) targets in this export set were already defined.\nTargets Defined: ${_targetsDefined}\nTargets not yet defined: ${_targetsNotDefined}\n")
│ │  endif()
│ │  unset(_targetsDefined)
│ │  unset(_targetsNotDefined)
│ │  unset(_expectedTargets)
│ │  
│ │  
│ │ -# Compute the installation prefix relative to this file.
│ │ -get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -if(_IMPORT_PREFIX STREQUAL "/")
│ │ -  set(_IMPORT_PREFIX "")
│ │ -endif()
│ │ +# The installation prefix configured by this project.
│ │ +set(_IMPORT_PREFIX "/usr")
│ │  
│ │  # Create imported target nlohmann_json::nlohmann_json
│ │  add_library(nlohmann_json::nlohmann_json INTERFACE IMPORTED)
│ │  
│ │  set_target_properties(nlohmann_json::nlohmann_json PROPERTIES
│ │    INTERFACE_COMPILE_DEFINITIONS "JSON_USE_IMPLICIT_CONVERSIONS=\$<BOOL:ON>"
│ │    INTERFACE_COMPILE_FEATURES "cxx_std_11"
│ ├── usr/lib/pkgconfig/nlohmann_json.pc
│ │ @@ -1,4 +1,4 @@
│ │  Name: nlohmann_json
│ │  Description: JSON for Modern C++
│ │  Version: 3.9.0
│ │ -Cflags: -I/usr/local/include
│ │ +Cflags: -I/usr/include

When not running the tests the package is reproducible and the above discrepancy can not be found.

Which compiler and operating system are you using?

• Compiler: gcc 10.1.0
• Operating system: Arch Linux
• cmake: 3.18.0

Which version of the library did you use?

• latest release version 3.9.0
• other release - please state the version: ___
• the develop branch

If you experience a compilation error: can you [compile and run the unit tests]

n/a

[nlohmann]

Could you please try with the latest develop version - we recently fixed two bugs introduced with the pkg-config file: #2314 and #2318. This is just a guess.

[dvzrv]

@nlohmann #2318 is applied, as otherwise installation would fail after running the tests.

[nlohmann]

So the problem is that the pkg-config and the cmake file contains the name of the binary directory?

[dvzrv]

The problem is, that the tests change the contents of those files. This is even the case, if I make a copy of the build directory (e.g. cp -av build build-test) and run ctest in the copy (i.e. in build-test).
This should not happen.

[nlohmann]

Is there a way I can reproduce this without Arch Linux?

[dvzrv]

I found a solution that is not pretty at all, but gets around the problem of the tests interfering with the packaged files:
I am now building twice (once for the installation, once for the tests) and then run the tests against the test build.

AFAICS the tests output things into the destination/output directory which overwrites previously set configuration and/or alters previously set configuration directly (see mention of nlohmann_json.pc in the initial comment). This has also been mentioned in #2318.

Test data should never leak into whatever is installed to the system!

Is there a way I can reproduce this without Arch Linux?

I'm not entirely sure. Other distributions also offer reproducible builds tools. I can't really provide any guidance for other distributions unfortunately.

Apart from that you could look at the nlohmann_json.pc file when setting CMAKE_INSTALL_PREFIX to '/usr'. When not running the tests and installing, it will correctly set the Cflags to -I/usr/include. If the tests are run, the output directory is overwritten with test data and Cflags is set to -I/usr/local/include (ouf).

If you want to try Arch Linux in a virtual machine, we provide it as a vagrant box:

vagrant init archlinux/archlinux
vagrant up
vagrant ssh

In the VM, you can prepare the system for building packages, import your own GPG public key and get the PKGBUILD for the package:

sudo pacman -Syu archlinux-repro asp base-devel devtools
gpg --recv-keys '797167AE41C0A6D9232E48457F3CEA63AE251B69'
asp checkout nlohmann-json
cd nlohmann-json/trunk

Here you'd have to modify the PKGBUILD to again run the tests against the initial build directory (i.e. change to make -k test -C build in the check() function).
Then you can build the package and run repro on it:

extra-x86_64-build
repro -ndf nlohmann-json-3.9.0-1-any.pkg.tar.zst

[nlohmann]

I found a solution that is not pretty at all, but gets around the problem of the tests interfering with the packaged files:
I am now building twice (once for the installation, once for the tests) and then run the tests against the test build.

AFAICS the tests output things into the destination/output directory which overwrites previously set configuration and/or alters previously set configuration directly (see mention of nlohmann_json.pc in the initial comment). This has also been mentioned in #2318.

Looking at #2318, it seems that the test cmake_add_subdirectory is the culprit then. Is everything fine when you skip the CMake-based tests then? (just comment the last 5 lines from tests/CMakeLists.txt.

[dvzrv]

Yes, removing the add_subdirectory() calls from the file makes the package reproducible as well.

[nlohmann]

@dvzrv Can you please check whether 1047d09 fixes the issue? In particular, whether ctest -LE not_reproducible does the job?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.