Skip to content

Cascading failure on bots when interrupted when performing git operations #34595

Closed

Description

This is basically a best guess as to what's happening on the bots, but from time to time it appears that the build directory gets entirely corrupted with error messages like:

extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rust-std-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rustc-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-22\cargo-nightly-x86_64-pc-windows-msvc.tar.gz
Synchronizing submodule url for 'src/compiler-rt'
Synchronizing submodule url for 'src/jemalloc'
Synchronizing submodule url for 'src/liblibc'
Synchronizing submodule url for 'src/llvm'
Synchronizing submodule url for 'src/rt/hoedown'
Synchronizing submodule url for 'src/rust-installer'
fatal: Needed a single revision


command did not execute successfully: "git" "submodule" "update"
expected success, got: exit code: 1


Makefile:23: recipe for target 'all' failed
>> rustjob: found 0 remaining processes
Unable to find current revision in submodule path 'src/llvm'
make: *** [all] Error 1

This happened on a MSVC rustbuild bot, but I believe I've seen this before on basically all bots (but I feel the windows bots are affected more often).

My best guess as to what's happening here is:

  • A build is running, and it's updating the LLVM submodule
  • The build is canceled for any number of a few reasons
  • The LLVM update process is interrupted, leaving the git directory in a corrupt state
  • All future builds attempt to use this corrupt directory and fail

I... don't actually know what the best solution here is. May as well track it though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions