Farmer uses a lot of RAM

[Neverhood13]

Node is looped on one block and then fills all the RAM staying on the same block.

aa3.mp4

[nazar-pc]

Looks like too many CPU cores for the amount of RAM causes too much parallel work happening on the farmer that leads to excessive RAM usage.

@i1i1 please try to reproduce this and see what can be done about it. We have a bunch of places in the farmer that start parallel work without any backpressure.

@mrSinisterE can you share if you're using HDD or SSD (SATA or NVMe) and how much space did you allocate for farmer?

[Neverhood13]

@nazar-pc HDD, 750G

[nazar-pc]

Thanks for report, @mrSinisterE!
We'll take a look at this in more detail in coming days.

[i1i1]

@mrSinisterE, thanks for all your input. It would be helpful if you could provide your cpu info. Just run lscpu and paste output here

[nazar-pc]

You can see it is 20 cores with 16G of RAM, doesn't matter what CPU too much I think

[i1i1]

I just think that total number of cores/threads is hidden by the window? Anyway, will test for 20 cores setup

[Neverhood13]

10 cores, 20 threads

[i1i1]

Hmm, 🤔 can't reproduce this issue. Tried following:

• 60M of ram were used with benching command
• 200M were used during sync

I seted up 20 threads for rayon, so it should be close to that. 🤷. Also plot size was set to 750G.

Will try syncing for a bit longer

[Neverhood13]

Hmm, 🤔 can't reproduce this issue. Tried following:

• 60M of ram were used with benching command
• 200M were used during sync

I seted up 20 threads for rayon, so it should be close to that. 🤷. Also plot size was set to 750G.

Will try syncing for a bit longer

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 20
On-line CPU(s) list: 0-19
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
Stepping: 2
CPU MHz: 1198.094
CPU max MHz: 3500.0000
CPU min MHz: 1200.0000
BogoMIPS: 5786.80
Virtualization: VT-x
L1d cache: 320 KiB
L1i cache: 320 KiB
L2 cache: 2.5 MiB
L3 cache: 25 MiB
NUMA node0 CPU(s): 0-19
Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

At the time the video was recorded, the size of the plot was almost completely full.

[nazar-pc]

@i1i1 I think this might be related to recommitments according to forum thread. Try to plot a bunch of stuff and they run recommitment over it (on proper plot just stopping farmer, deleting commitments databases and starting again will work).

[alphatangoalpha]

This is fairly easy to reproduce. Plot 750G fully, wait for salt update and RAM requirements slowly go up to 32G per farmer process. Iscpu output below.

]# dmesg
[ 4559.591900] Out of memory: Killed process 473322 (subspace-farm) total-vm:30285908kB, anon-rss:19406912kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:41124kB oom_score_adj:1000
[ 5015.602498] Out of memory: Killed process 499067 (subspace-farm) total-vm:32024100kB, anon-rss:21638096kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:44036kB oom_score_adj:1000
[ 5503.813502] Out of memory: Killed process 501754 (subspace-farm) total-vm:35191672kB, anon-rss:24674596kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:50244kB oom_score_adj:1000
[ 6287.953751] Out of memory: Killed process 509618 (subspace-farm) total-vm:38799288kB, anon-rss:28334516kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:57232kB oom_score_adj:1000
[ 9274.259649] Out of memory: Killed process 521986 (subspace-farm) total-vm:44653580kB, anon-rss:33661600kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:68972kB oom_score_adj:1000

lscpu

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 40
On-line CPU(s) list: 0-39
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
Stepping: 4
CPU MHz: 1951.551
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000
BogoMIPS: 4389.56
Virtualization: VT-x
L1d cache: 640 KiB
L1i cache: 640 KiB
L2 cache: 5 MiB
L3 cache: 50 MiB
NUMA node0 CPU(s): 0-9,20-29
NUMA node1 CPU(s): 10-19,30-39
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopolo
gy nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_faul
t epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d

[i1i1]

@nazar-pc tried recommitment bench on 750G plot, and it consumes around 750-900M of ram. Will try to sync node with such setup

[alphatangoalpha]

Here is a stack trace while it was consuming 25G

Thread 24 (Thread 0x7f2b8f9fc000 (LWP 310) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560796e2f8aa in rayon_core::sleep::Sleep::sleep ()
#3  0x0000560796e2f34b in rayon_core::registry::WorkerThread::wait_until_cold ()
#4  0x00005607972458dc in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x000056079724545f in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 23 (Thread 0x7f2b537fe000 (LWP 309) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560796e2f8aa in rayon_core::sleep::Sleep::sleep ()
#3  0x0000560796e2f34b in rayon_core::registry::WorkerThread::wait_until_cold ()
#4  0x00005607972458dc in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x000056079724545f in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 22 (Thread 0x7f2b517fa000 (LWP 308) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560796e2f8aa in rayon_core::sleep::Sleep::sleep ()
#3  0x0000560796e2f34b in rayon_core::registry::WorkerThread::wait_until_cold ()
#4  0x0000560796eb9167 in rayon_core::join::join_context::{{closure}} ()
#5  0x0000560797015940 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#6  0x00005607970726de in <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute ()
#7  0x0000560796e2f278 in rayon_core::registry::WorkerThread::wait_until_cold ()
#8  0x000056079741c037 in rayon_core::join::join_context::{{closure}} ()
#9  0x000056079741a788 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#10 0x000056079741bfce in rayon_core::join::join_context::{{closure}} ()
#11 0x000056079741a788 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#12 0x0000560796f38151 in subspace_farmer::plotting::plot_pieces::{{closure}} ()
#13 0x00005607970157fa in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#14 0x0000560796eb9013 in rayon_core::join::join_context::{{closure}} ()
#15 0x0000560797015940 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#16 0x00005607970726de in <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute ()
#17 0x0000560796e2f278 in rayon_core::registry::WorkerThread::wait_until_cold ()
--Type <RET> for more, q to quit, c to continue without paging--c
#18 0x00005607972458dc in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#19 0x000056079724545f in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#20 0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#21 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#22 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#23 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#24 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 21 (Thread 0x7f2b50ff9000 (LWP 307) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797395414 in subspace_farmer::plot::Plot::write_many ()
#2  0x0000560796f381f2 in subspace_farmer::plotting::plot_pieces::{{closure}} ()
#3  0x00005607970157fa in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#4  0x00005607970726de in <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute ()
#5  0x0000560796e2f278 in rayon_core::registry::WorkerThread::wait_until_cold ()
#6  0x0000560796eb9167 in rayon_core::join::join_context::{{closure}} ()
#7  0x0000560797015940 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#8  0x0000560796eb9013 in rayon_core::join::join_context::{{closure}} ()
#9  0x0000560797015940 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#10 0x00005607970726de in <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute ()
#11 0x0000560796e2f278 in rayon_core::registry::WorkerThread::wait_until_cold ()
#12 0x00005607972458dc in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#13 0x000056079724545f in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#14 0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#15 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#16 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#17 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 20 (Thread 0x7f2b23fff000 (LWP 306) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560796e2f8aa in rayon_core::sleep::Sleep::sleep ()
#3  0x0000560796e2f34b in rayon_core::registry::WorkerThread::wait_until_cold ()
#4  0x0000560796eb9167 in rayon_core::join::join_context::{{closure}} ()
#5  0x0000560797015940 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#6  0x0000560796eb9013 in rayon_core::join::join_context::{{closure}} ()
#7  0x000056079707253c in <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute ()
#8  0x0000560796e2f278 in rayon_core::registry::WorkerThread::wait_until_cold ()
#9  0x00005607972458dc in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#10 0x000056079724545f in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#11 0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#12 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#13 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#14 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 19 (Thread 0x7f2b52ffd000 (LWP 290) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x00005607973452ff in std::sys_common::thread_parker::futex::Parker::park_timeout () at library/std/src/sys_common/thread_parker/futex.rs:72
#3  std::thread::park_timeout () at library/std/src/thread/mod.rs:993
#4  0x0000560797493363 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797492845 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 18 (Thread 0x7f2b527fc000 (LWP 259) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 17 (Thread 0x7f2b51ffb000 (LWP 228) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 16 (Thread 0x7f2ae9fcb000 (LWP 197) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 15 (Thread 0x7f2b8fbfd000 (LWP 91) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 14 (Thread 0x7f2b8ffff000 (LWP 89) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x7f2b75ffb000 (LWP 88) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x00005607972443b2 in rayon_core::latch::LockLatch::wait_and_reset ()
#3  0x0000560796e0249c in rayon_core::registry::Registry::in_worker_cold ()
#4  0x00005607970159d7 in rayon::iter::plumbing::bridge_producer_consumer::helper ()
#5  0x000056079702f363 in tokio::runtime::task::raw::poll ()
#6  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#7  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#8  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#9  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#11 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 12 (Thread 0x7f2b757fa000 (LWP 87) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x0000560797345249 in std::sys_common::thread_parker::futex::Parker::park () at library/std/src/sys_common/thread_parker/futex.rs:52
#3  std::thread::park () at library/std/src/thread/mod.rs:929
#4  0x0000560796f9c425 in std::sync::mpsc::sync::wait ()
#5  0x0000560796f9a528 in std::sync::mpsc::Receiver<T>::recv ()
#6  0x0000560797025715 in tokio::runtime::task::raw::poll ()
#7  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#10 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#11 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#12 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x7f2b74ff9000 (LWP 86) "tokio-runtime-w"):
#0  __GI___libc_write (nbytes=8, buf=0x7f2b74ff4380, fd=38) at ../sysdeps/unix/sysv/linux/write.c:26
#1  __GI___libc_write (fd=38, buf=0x7f2b74ff4380, nbytes=8) at ../sysdeps/unix/sysv/linux/write.c:24
#2  0x000056079739de9b in std::io::Write::write_all ()
#3  0x0000560797026ada in tokio::runtime::task::raw::poll ()
#4  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7f2b53fff000 (LWP 85) "tokio-runtime-w"):
#0  __futex_abstimed_wait_common64 (private=480202, cancel=true, abstime=0x7f2b53ffa7a0, op=393, expected=0, futex_word=0x7f2b9c003610) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=480202, abstime=0x7f2b53ffa7a0, clockid=1744839968, expected=0, futex_word=0x7f2b9c003610) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f2b9c003610, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2b53ffa7a0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f2bb612bf1b in __pthread_cond_wait_common (abstime=0x7f2b53ffa7a0, clockid=0, mutex=0x7f2b9c0035a8, cond=0x7f2b9c0035e8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x7f2b9c0035e8, mutex=0x7f2b9c0035a8, abstime=0x7f2b53ffa7a0) at ./nptl/pthread_cond_wait.c:652
#5  0x00005607976f9ed3 in rocksdb::port::CondVar::TimedWait(unsigned long) ()
#6  0x00005607976b563d in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) ()
#7  0x00005607975ee34c in rocksdb::Timer::Run() ()
#8  0x00007f2bb64b42c3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7f2bb4a6c000 (LWP 69) "rocksdb:high"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5607984425e8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5607984425e8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5607984425e8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f2bb612bac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x560798442598, cond=0x5607984425c0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5607984425c0, mutex=0x560798442598) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f2bb64847bd in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00005607977b08af in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#7  0x00005607977b0c57 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#8  0x00007f2bb64b42c3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7f2bb526d000 (LWP 68) "rocksdb:low"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5607984424ac) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5607984424ac) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5607984424ac, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f2bb612bac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x560798442458, cond=0x560798442480) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x560798442480, mutex=0x560798442458) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f2bb64847bd in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00005607977b08af in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#7  0x00005607977b0c57 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#8  0x00007f2bb64b42c3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f2bb5675000 (LWP 66) "futures-timer"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000560797343c70 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:61
#2  0x00005607973452ff in std::sys_common::thread_parker::futex::Parker::park_timeout () at library/std/src/sys_common/thread_parker/futex.rs:72
#3  std::thread::park_timeout () at library/std/src/thread/mod.rs:993
#4  0x000056079718ccec in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x000056079718c215 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f2bb587c000 (LWP 65) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056079723cb89 in parking_lot::condvar::Condvar::wait_until_internal ()
#2  0x0000560797435674 in tokio::runtime::thread_pool::worker::Context::park_timeout ()
#3  0x00005607974336c0 in tokio::runtime::task::raw::poll ()
#4  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f2bb5a80000 (LWP 64) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056079723cb89 in parking_lot::condvar::Condvar::wait_until_internal ()
#2  0x0000560797435674 in tokio::runtime::thread_pool::worker::Context::park_timeout ()
#3  0x00005607974336c0 in tokio::runtime::task::raw::poll ()
#4  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f2bb5c84000 (LWP 63) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056079723cb89 in parking_lot::condvar::Condvar::wait_until_internal ()
#2  0x0000560797435674 in tokio::runtime::thread_pool::worker::Context::park_timeout ()
#3  0x00005607974336c0 in tokio::runtime::task::raw::poll ()
#4  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f2bb5e85000 (LWP 62) "tokio-runtime-w"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056079723cb89 in parking_lot::condvar::Condvar::wait_until_internal ()
#2  0x0000560797435674 in tokio::runtime::thread_pool::worker::Context::park_timeout ()
#3  0x00005607974336c0 in tokio::runtime::task::raw::poll ()
#4  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#7  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#9  0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f2bb6089000 (LWP 61) "tokio-runtime-w"):
#0  0x00007f2bb61bdfde in epoll_wait (epfd=3, events=0x560798467b30, maxevents=1024, timeout=948) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000560797423dfb in tokio::io::driver::Driver::turn ()
#2  0x000056079742bf35 in tokio::time::driver::Driver<P>::park_internal ()
#3  0x00005607974356e2 in tokio::runtime::thread_pool::worker::Context::park_timeout ()
#4  0x00005607974336c0 in tokio::runtime::task::raw::poll ()
#5  0x0000560797428d57 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#6  0x0000560797428ab0 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#7  0x000056079736b5d5 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#8  alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007f2bb612cb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f2bb61bea00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f2bb6093000 (LWP 20) "subspace-farm"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056079723cb89 in parking_lot::condvar::Condvar::wait_until_internal ()
#2  0x0000560797422283 in tokio::park::thread::Inner::park ()
#3  0x00005607970e5d3c in subspace_farmer::main ()
#4  0x0000560796f8ee93 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x00005607970f4db3 in main ()

[nazar-pc]

Found the reason, working on solution now

[nazar-pc]

Tested fix from upcoming release, used ~400M of RAM on the farmer with 300G plot size. I think this issue will be resolved.

[hung-nguyen89]

I'm not sure it is same issue, but farmer using 20G+ RAM with 2TB plot
version subspace-farmer-windows-x86_64-skylake-gemini-3f-2023-sep-25.exe

I found some discussion on Discord, said that Windows report wrong RAM usage.
I don't think that the case, because after use all RAM, Windows start to use pagefile, then system become laggy, alot of glitch.

[nazar-pc]

I'm fairly certain it is not. Please create a forum thead with additional information like screenshot of process list that shows how much RAM farmer uses. This resolved issue is not the best place for such discussions.