Memory usage regression since 5/1

[ishitatsuyuki]

It's back, caused by presumably #50197.

cc @nikomatsakis

   Compiling syntax_ext v0.0.0 (file:///home/user/rust/src/libsyntax_ext)
error: could not exec the linker `cc`
  |
  = note: Cannot allocate memory (os error 12)
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-c6263d330d2b9f31.rustc0-81414edc9fa898f39b569bc75d0a157e.rs.rcgu.o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-c6263d330d2b9f31.rustc1-81414edc9fa898f39b569bc75d0a157e.rs.rcgu.o" "-o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc-c6263d330d2b9f31.so" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-c6263d330d2b9f31.crate.metadata.rcgu.o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-c6263d330d2b9f31.crate.allocator.rcgu.o" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-efe11c48d393cb3a/out/.libs" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/miniz-sys-ed3f79652d61a4a4/out" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "test-902559d4a0cc83d5" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "term-36880752958d2c6f" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libflate2-be463fa6b96a16ec.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libminiz_sys-faafc4fdd43b8a1b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libbacktrace-447434a17b628693.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libbacktrace_sys-63d1b058c70a5ba5.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/librustc_demangle-ffb675b42151a6f8.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libbyteorder-f652c706c62d3d96.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "proc_macro-c7d45a460cb3a738" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/libjobserver-5189829dca651532.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "rustc_const_math-e212b3283c198d45" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "syntax-e97e6d4ccc983706" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_errors-dafdecc781917b18" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "syntax_pos-ccbedf94f6bd555f" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_target-aaec33fa4a23ec51" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.BU4zySi8u8L1/liblazy_static-d0629a75f2783580.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "graphviz-06bb7008d45cdff0" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "fmt_macros-7fa0522df5af949a" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "arena-16294da79648b864" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_data_structures-22e458fc7e0d421f" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_cratesio_shim-7bdcd669c7579602" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "serialize-b63c42e100577192" "-Wl,--start-group" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "std-b63203ee1146648a" "-Wl,--end-group" "-Wl,-Bstatic" "/tmp/rustc.BU4zySi8u8L1/libcompiler_builtins-bbf524080a1524b2.rlib" "-Wl,-Bdynamic" "-l" "util" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util" "-shared" "-Wl,-rpath,$ORIGIN/../lib"

error: aborting due to previous error

error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully: `/home/user/rust/build/bootstrap/debug/rustc --crate-name rustc librustc/lib.rs --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 -C metadata=c6263d330d2b9f31 -C extra-filename=-c6263d330d2b9f31 --out-dir /home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern rustc_target=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-aaec33fa4a23ec51.so --extern proc_macro=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libproc_macro-c7d45a460cb3a738.so --extern byteorder=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-f652c706c62d3d96.rlib --extern lazy_static=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblazy_static-d0629a75f2783580.rlib --extern rustc_apfloat=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-262722bdf7d51c3a.rlib --extern bitflags=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-fd8a318e919a8b90.rlib --extern jobserver=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libjobserver-5189829dca651532.rlib --extern graphviz=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-06bb7008d45cdff0.so --extern rustc_const_math=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_const_math-e212b3283c198d45.so --extern rustc_data_structures=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-22e458fc7e0d421f.so --extern fmt_macros=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libfmt_macros-7fa0522df5af949a.so --extern log=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-19a330a6ee3947d3.rlib --extern rustc_errors=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-dafdecc781917b18.so --extern syntax_pos=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-ccbedf94f6bd555f.so --extern arena=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-16294da79648b864.so --extern tempdir=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libtempdir-83e93e20d067e6f5.rlib --extern syntax=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-e97e6d4ccc983706.so --extern backtrace=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbacktrace-447434a17b628693.rlib --extern flate2=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libflate2-be463fa6b96a16ec.rlib --extern serialize=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-b63c42e100577192.so --extern serialize=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-b63c42e100577192.rlib -L native=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-efe11c48d393cb3a/out/.libs -L native=/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/miniz-sys-ed3f79652d61a4a4/out` (exit code: 101)

[ishitatsuyuki]

Although I'm not absolutely confident as I haven't performed a bisect, and perf data is not updated yet as well.

[nikomatsakis]

@ishitatsuyuki

It's back

What exactly are you doing to reproduce this? Is that a reference to an earlier issue? (I vaguely remember an issue about this maybe...?)

[ishitatsuyuki]

@nikomatsakis It's a reference to #50073. Basically, I mean that the OOM error happened again on that environment:

On the CircleCI Docker remote machine which has 2 cores and 8G memory, I'm no longer able to compile rustc. This error consistently occurs in stage1.

[nikomatsakis]

@ishitatsuyuki thanks! perf is still not updated, it seems. #50197 could certainly be the cause, that could increase the size of the interning tables.

[nikomatsakis]

triage: P-high -- we're investigating. First thing is to get the perf server up and going again, probably. ( cc @Mark-Simulacrum )

[alexcrichton]

Scanning the current max-rss statistics of perf.rust-lang.org doesn't show much that's suspicious (no recent spikes up)

[ishitatsuyuki]

What I currently know is that compiling stage1 is taking much more memory at peak, compared to stage0. It's like 5G vs 3G.

[ishitatsuyuki]

As of today it seems to compile successfully; so this is no longer critical. I will reopen if this happens again.