4 pack-related tests fail on s390x #2816
Replies: 1 comment
|
Sorry I haven't picked this back up for so long. I had an IBM Cloud credit, which I was able to use to work on this, and if I recall correctly, I completed testing it, except for some analysis. Then the GitHub Secure Open Source Fund happened, and I put this on hold. Doing that instead of finishing this up was a super worthwhile tradeoff! But I have been meaning to return to this. I should be able to finish the analysis and recheck; if there is no apparent progress on week, then you or anyone should feel free to ping me Eliah Kagan (@EliahKagan). I think it's not at all reasonable for me to ask that anyone else pull material from what I figured out so far, but if you want to do so, the gists are part one, part two, part three, part four, and a few others that are less important and can be ignored. The failures seem to come down to which shared library is used for something (sorry about the vagueness; I don't have time to reread what I wrote there right now). There may or may not be a reasonable step to take on our end to improve it; I hope to come to a conclusion about that when I pick this back up, if no one else ends up doing so first. |
Uh oh!
There was an error while loading. Please reload this page.
Current behavior 😯
On an s390x system running Ubuntu 22.04 LTS with git 2.48.1 installed from the git-core PPA, two tests fail when using committed fixture archives, and those tests as well as two others fail when ignoring archives.
Full test output of the most recent runs can be viewed in this gist. The same failures have been happening for a while; that gist links to a few older gists. This is not new as of git 2.48.1; it is the same problem as observed with git 2.47.0 (also installed through the git-core PPA) as mentioned in #1687 and discusses in comments in #1698, starting at #1698 (comment).
Without
GIX_TEST_IGNORE_ARCHIVES cargo nextest run --workspace --no-fail-fast, the failing tests are:With
GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --workspace --no-fail-fast, the failing tests are:Notably, all these failing tests relate to packs. But various other tests related to packs do not fail.
Rerunning the first two failing tests (those that occur even when using committed archives) by themselves and enabling a backtrace shows:
Rerunning the two tests that fail only when ignoring commited archives shows:
Expected behavior 🤔
All tests should pass. (Unless it turns out that the failures are due to entirely to bugs in
gititself.)s390x is bigendian, and to the best of my knowledge is the most prominent bigendian architecture for general-purpose use. Bugs that occur specifically on s390x suggest the possibility of unsoundness, though they could arise without any soundness bugs if they happen due to a mismatch in expectations in how data structures are written to disk on different architectures. (If the representation is not validated before being written into memory using unsafe code, then that would itself be unsound, but that would be unsound even in the absence of a mismatch in expectations.)
Git behavior
Presumably subtleties about
gitbehavior are relevant, since two of the failures only occur when rerunning fixture scripts, but I do not know how. This need not imply that a bug in thegitis the cause.Steps to reproduce 🕹
On Ubuntu 22.04 LTS on s390x, I enabled the git-core PPA and upgraded
git. I verified the git version withgit versionand by runningapt list gitto see that it is installed from the PPA.I ran the test suite, allowing committed archives to be used, showing the first two failures:
I did a full clean, then ran the test suite again, forcing it to rerun fixture scripts, showing those failures again as well as the last two:
I restored the original archives, removed generated repositories, and ran the first two failing tests with backtraces:
I removed generated repositories again and ran the other two failing tests with backtraces:
The full workspace runs without backtraces are shown in the above-linked gist, with older such runs shown in other gists that it links to. The specific runs' full output are shown above in the "Current behavior" section.
I have not been able to reproduce it with
crossWe do not test s390x on CI. Running the full test suite in s390x on CI may not be justifiable, because GitHub Actions does not support s390x, and tests would run slower under QEMU emulation. However, this is feasible, including through
cross, which speeds things up by doing the actual building with tools that themselves can run without emulation. (ThecrossDocker images for non-x86-64 targets are actually x86-64 images, which have toolchains on them that are built for x86-64 but that build the other non-x86-64 target. It is also possible to install dependencies on them for the cross-target architecture, which can run in QEMU via binfmt_misc, and sometimes they contain some such dependencies, such as the Android NDK on Android images.) More likely to be justifiable would be to run a subset of the tests, including any tests that have failed before or about which there is specific concern.The testing described above was done on actual s390x hardware. (Specifically, it was done without containerization, on an s390x virtual machine, where the host was also s390x and hardware virtualization was used.) But I have also attempted to use
crossfor this. When I do, I have so far been unable to produce these failures--all s390x tests pass. This happens both withmax-pureand with all default features, and includes all the tests that fail here. They pass in the container.The version of
giton thecrossimage is old, such that tests will fail if it is not upgraded. I upgrade it, also using the git-core PPA added as a repository in thecrossimage. This allows all tests to pass in the image, both withGIX_TEST_IGNORE_ARCHIVESunset and with it set to1. (By default,crossfilters most environment variables from outside the container. I account for this and verify that the environment variables are present and have their expected effects.)The tests pass when amd64 (i.e. x86-64) builds of
gitand its dependencies are installed from the git-core PPA. They also pass when I instead install s390x builds (which the git-core PPA also provides) ofgitand s390x builds of its library dependencies and of its non-library dependencies, except that I am not able to install s390x builds ofperlandliberror-perl, so I instead force s390xgitto use the amd64perlandliberror-perlby tellingdpkgto allow those s390x dependencies to be unsatisfied and ensuring the amd64 packages are installed.gitis able to useperlof a different architecture because it uses it to run scripts, rather than linking to it as a shared library (and whileliberror-perlis a library package, it provides a Perl module, not a shared library thatgititself links to). Since I don't think recent versions ofgituseperlvery heavily, it seems unlikely that the architectureperlis built for is what makes the difference here, but I have not ruled that out.I have verified that the failures occur, and in the same way, on the system where I have observed them, even when I use
cargo testrather thancargo nextest run. This is relevant tocrossbecause cross does not support runningnextest; when usingcross, I have been usingcross test, which runscargo testin the container. Differences betweennextestandcargo-testare not the cause. (This is worth checking becausenextestmakes greater use of multiprocessing to run multiple tests at a time, whilecargo-testmakes greater use of multithreading. However, in this case, the tests fail even when run individually--which I have also verified.)Originally tracked by Eliah Kagan (@EliahKagan) in issue #1890.
All reactions