ICE: unstable fingerprints for native_libraries

[kostko]

Code

Compiling oasis-core with nightly 2021-05-06 and incremental compilation enabled. Seems to happen during compilation of grpcio v0.8.2.

Meta

rustc --version --verbose:

rustc 1.54.0-nightly (676ee1472 2021-05-06)
binary: rustc
commit-hash: 676ee14729462585b969bbc52f32c307403f4126
commit-date: 2021-05-06
host: x86_64-unknown-linux-gnu
release: 1.54.0-nightly
LLVM version: 12.0.0

Error output

   Compiling oasis-core-runtime v0.3.0-alpha (/workdir/runtime)
   Compiling oasis-core-keymanager-api-common v0.3.0-alpha (/workdir/keymanager-api-common)
   Compiling grpcio v0.8.2
   Compiling oasis-core-client v0.3.0-alpha (/workdir/client)
   Compiling oasis-core-keymanager-client v0.3.0-alpha (/workdir/keymanager-client)
thread 'rustc' panicked at 'assertion failed: `(left == right)`
  left: `Some(Fingerprint(3744085167471343643, 10066438777687008338))`,
 right: `Some(Fingerprint(10047237832729537343, 9265653581121216388))`: found unstable fingerprints for native_libraries(grpcio_sys[1373]): [NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_civil_time"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_throw_delegate"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_city"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_time"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_raw_hash_set"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_bad_variant_access"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("upb"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("re2"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_str_format_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_hashtablez_sampler"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_log_severity"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_hash"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_graphcycles_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_synchronization"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_bad_optional_access"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("address_sorting"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_int128"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("z"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_strings"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_statusor"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_symbolize"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("gpr"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_stacktrace"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_debugging_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("grpc"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_exponential_biased"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_raw_logging_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_status"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("cares"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_demangle_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_base"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_malloc_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_time_zone"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_cord"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_strings_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_spinlock_wait"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("ssl"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("crypto"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("grpc_wrap"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Unspecified, name: Some("stdc++"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }]', /rustc/676ee14729462585b969bbc52f32c307403f4126/compiler/rustc_query_system/src/query/plumbing.rs:585:5
stack backtrace:
   <snip>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
 
error: internal compiler error: unexpected panic
 
note: the compiler unexpectedly panicked. this is a bug.
 
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
 
note: rustc 1.54.0-nightly (676ee1472 2021-05-06) running on x86_64-unknown-linux-gnu
 
note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental -C target-feature=+aes,+ssse3 --crate-type lib
 
note: some of the compiler flags provided by cargo are hidden
 
query stack during panic:
#0 [native_libraries] looking up the native libraries of a linked crate
end of query stack

Backtrace

   0: rust_begin_unwind
             at /rustc/676ee14729462585b969bbc52f32c307403f4126/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/676ee14729462585b969bbc52f32c307403f4126/library/core/src/panicking.rs:92:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
   4: rustc_query_system::query::plumbing::incremental_verify_ich
   5: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   6: rustc_query_system::query::plumbing::get_query_impl
   7: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::native_libraries
   8: rustc_codegen_ssa::base::<impl rustc_codegen_ssa::CrateInfo>::new
   9: rustc_codegen_ssa::back::write::start_async_codegen
  10: rustc_codegen_ssa::base::codegen_crate
  11: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  12: rustc_interface::passes::QueryContext::enter
  13: rustc_interface::queries::Queries::ongoing_codegen
  14: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  15: rustc_span::with_source_map
  16: rustc_interface::interface::create_compiler_and_run
  17: scoped_tls::ScopedKey<T>::set

[Aaron1011]

@kostko: Looking at the log in https://buildkite.com/oasisprotocol/oasis-core-ci/builds/4881#_, I can only see oasis-core-client being built once. For this crash to occur, there needs to be an existing target/debug/incremental or target/release/incremental directory. Is your CI setup performing any kind of caching across runs? If so, can you link me to the code that performs that?

[kostko]

Yes, it uses caching (used until we disabled it in the PR linked above) by sharing the incremental compile caches across invocations.

[Aaron1011]

@kostko: I'm not familiar with Buildkit - does that share the incremental directories across different builds (e.g. of different PRs or commits), or just across the different steps (e.g. 'Lint Rust crate versions' and 'Build Rust runtime loader') in a build like https://buildkite.com/oasisprotocol/oasis-core-ci/builds/4881#_ ?

[kostko]

Oh so we have custom Buildkite agents and these builds are running in a Docker container where the incremental directories are bound to a common directory on the host (agent). So the cache is shared across different builds (e.g., different PRs/commits/etc.) that happen on the same agent not only across different steps.

[Aaron1011]

This issue should be fixed by #83901 (which isn't part of the 2021-05-06 nightly build). Try updating to the latest nightly - if the issue occurs again, let me know.

[kostko]

Cool, will try the most recent nightly.

[kostko]

Tried with f94942d84 2021-05-19 (and clearing the IC caches) but it still seems to hit the same problem (example).

[Aaron1011]

PR #85702 fixes another issue that affects native_libraries. However, it's possible that there are still more undiscovered issues affected native_libraries.

[Aaron1011]

The latest nightly contains a fix for another bug that affects native_libraries. @kostko Could you upgrade your Rust nightly again?