Repository navigation
Crashes on Fedora 32 on replay #2469
Description
It's a custom Fedora 32 build:
gdb output:
(gdb) bt
#0 0x00007f504fd70cdb in __GI___close_nocancel () from /lib64/ld-linux-x86-64.so.2
#1 0x00007f504fd5c45e in _dl_map_object_from_fd () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f504fd5d91b in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f504fd55469 in map_doit () from /lib64/ld-linux-x86-64.so.2
#4 0x00007f504fd6fcbe in _dl_catch_exception () from /lib64/ld-linux-x86-64.so.2
#5 0x00007f504fd6fd63 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#6 0x00007f504fd553f0 in do_preload () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f504fd563d4 in handle_preload_list () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f504fd598cf in dl_main () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f504fd6ec3b in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f504fd55fbc in _dl_start () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f504fd55118 in _start () from /lib64/ld-linux-x86-64.so.2
#12 0x0000000000000002 in ?? ()
#13 0x00007ffdbdaff924 in ?? ()
#14 0x00007ffdbdaff932 in ?? ()
#15 0x0000000000000000 in ?? ()
rr output:
(rr) c
Continuing.
[ERROR /home/komat/tmp6/rr/src/TraceStream.cc:1073:read_mapped_region()] Metadata of /usr/lib64/rr/librrpreload.so changed: replay divergence likely, but continuing anyway. inode: 5377172/5256379; mode: 33261/33261; uid: 0/0; gid: 0/0; size: 608816/172208; mtime: 1585046706/1582565663
[ERROR /home/komat/tmp6/rr/src/TraceStream.cc:1073:read_mapped_region()] Metadata of /usr/lib64/rr/librrpreload.so changed: replay divergence likely, but continuing anyway. inode: 5377172/5256379; mode: 33261/33261; uid: 0/0; gid: 0/0; size: 608816/172208; mtime: 1585046706/1582565663
[ERROR /home/komat/tmp6/rr/src/TraceStream.cc:1073:read_mapped_region()] Metadata of /usr/lib64/rr/librrpreload.so changed: replay divergence likely, but continuing anyway. inode: 5377172/5256379; mode: 33261/33261; uid: 0/0; gid: 0/0; size: 608816/172208; mtime: 1585046706/1582565663
[ERROR /home/komat/tmp6/rr/src/TraceStream.cc:1073:read_mapped_region()] Metadata of /usr/lib64/rr/librrpreload.so changed: replay divergence likely, but continuing anyway. inode: 5377172/5256379; mode: 33261/33261; uid: 0/0; gid: 0/0; size: 608816/172208; mtime: 1585046706/1582565663
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] rbx 0x7ffdbdafcff0 != 0x6ffffdff (replaying vs. recorded)
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] rdx 0 != 0x7f504fd49f00 (replaying vs. recorded)
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] rsi 0x5360 != 0x29 (replaying vs. recorded)
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] r9 0 != 0xeffffef5 (replaying vs. recorded)
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] r10 0x4 != 0x70000022 (replaying vs. recorded)
[ERROR /home/komat/tmp6/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] r13 0x3 != 0x6ffffeff (replaying vs. recorded)
[FATAL /home/komat/tmp6/rr/src/Registers.cc:366:compare_register_files()]
(task 57135 (rec:31917) at time 56)
-> Assertion !bail_error || match' failed to hold. Fatal register mismatch (ticks/rec:11093/11155) Tail of trace dump: { real_time:16454.923568 global_time:36, event:SYSCALL: uname' (state:EXITING_SYSCALL) tid:31917, ticks:10448
rax:0x0 rbx:0x7f504fd816f0 rcx:0xffffffffffffffff rdx:0x78 rsi:0x7ffdbdb51120 rdi:0x7ffdbdafe6b0 rbp:0xc rsp:0x7ffdbdafe668 r8:0x70 r9:0x7f504fd76950 r10:0xffff8002424aeb5c r11:0x246 r12:0x0 r13:0x0 r14:0x0 r15:0x7f504fd81140 rip:0x7f504fd7111b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3f fs_base:0x0 gs_base:0x0
{ tid:31917, addr:0x7ffdbdafe6b0, length:0x186 }
}
{
real_time:16454.923599 global_time:37, event:INSTRUCTION_TRAP' tid:31917, ticks:10641 rax:0x2af7f13e rbx:0x7f504fd81040 rcx:0x7f504fd53270 rdx:0x26d1 rsi:0x0 rdi:0x7f504fd53000 rbp:0x7ffdbdafea40 rsp:0x7ffdbdafe860 r8:0x0 r9:0x7 r10:0x0 r11:0x8 r12:0x0 r13:0x0 r14:0x0 r15:0x7f504fd81140 rip:0x7f504fd598af eflags:0x10206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x0 gs_base:0x0 } { real_time:16454.923631 global_time:38, event:SYSCALL: openat' (state:ENTERING_SYSCALL) tid:31917, ticks:10778
rax:0xffffffffffffffda rbx:0x4000000 rcx:0xffffffffffffffff rdx:0x80000 rsi:0x7f504fd81d10 rdi:0xffffff9c rbp:0x7ffdbdafd250 rsp:0x7ffdbdafd1e8 r8:0x80000 r9:0x7f504fd81d10 r10:0x0 r11:0x246 r12:0x7ffdbdafd2af r13:0x7f504fd81d10 r14:0x7f504fd80000 r15:0x7ffdbdafd2c0 rip:0x7f504fd70e18 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.923682 global_time:39, event:SYSCALL: openat' (state:EXITING_SYSCALL) tid:31917, ticks:10778 rax:0x3 rbx:0x4000000 rcx:0xffffffffffffffff rdx:0x80000 rsi:0x7f504fd81d10 rdi:0xffffff9c rbp:0x7ffdbdafd250 rsp:0x7ffdbdafd1e8 r8:0x80000 r9:0x7f504fd81d10 r10:0x0 r11:0x246 r12:0x7ffdbdafd2af r13:0x7f504fd81d10 r14:0x7f504fd80000 r15:0x7ffdbdafd2c0 rip:0x7f504fd70e18 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x0 gs_base:0x0 } { real_time:16454.923714 global_time:40, event:SYSCALL: read' (state:ENTERING_SYSCALL) tid:31917, ticks:10780
rax:0xffffffffffffffda rbx:0x340 rcx:0xffffffffffffffff rdx:0x340 rsi:0x7ffdbdafd2c8 rdi:0x3 rbp:0x7ffdbdafd250 rsp:0x7ffdbdafd1e8 r8:0x80000 r9:0x7f504fd81d10 r10:0x0 r11:0x246 r12:0x7ffdbdafd2af r13:0x7f504fd81d10 r14:0x3 r15:0x7ffdbdafd2c0 rip:0x7f504fd70ed8 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x0 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.923742 global_time:41, event:SYSCALL: read' (state:EXITING_SYSCALL) tid:31917, ticks:10780 rax:0x340 rbx:0x340 rcx:0xffffffffffffffff rdx:0x340 rsi:0x7ffdbdafd2c8 rdi:0x3 rbp:0x7ffdbdafd250 rsp:0x7ffdbdafd1e8 r8:0x80000 r9:0x7f504fd81d10 r10:0x0 r11:0x246 r12:0x7ffdbdafd2af r13:0x7f504fd81d10 r14:0x3 r15:0x7ffdbdafd2c0 rip:0x7f504fd70ed8 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x0 fs_base:0x0 gs_base:0x0 { tid:31917, addr:0x7ffdbdafd2c8, length:0x340 } } { real_time:16454.923773 global_time:42, event:SYSCALL: fstat' (state:ENTERING_SYSCALL) tid:31917, ticks:10824
rax:0xffffffffffffffda rbx:0x4000000 rcx:0xffffffffffffffff rdx:0x7ffdbdafd170 rsi:0x7ffdbdafd170 rdi:0x3 rbp:0x7ffdbdafd230 rsp:0x7ffdbdafd0d8 r8:0x1 r9:0x7f504fd81140 r10:0x0 r11:0x246 r12:0x7f504fd81140 r13:0x7f504fd81140 r14:0x7f504fd80000 r15:0x3 rip:0x7f504fd70be9 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.923796 global_time:43, event:SYSCALL: fstat' (state:EXITING_SYSCALL) tid:31917, ticks:10824 rax:0x0 rbx:0x4000000 rcx:0xffffffffffffffff rdx:0x7ffdbdafd170 rsi:0x7ffdbdafd170 rdi:0x3 rbp:0x7ffdbdafd230 rsp:0x7ffdbdafd0d8 r8:0x1 r9:0x7f504fd81140 r10:0x0 r11:0x246 r12:0x7f504fd81140 r13:0x7f504fd81140 r14:0x7f504fd80000 r15:0x3 rip:0x7f504fd70be9 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0x0 gs_base:0x0 { tid:31917, addr:0x7ffdbdafd170, length:0x90 } } { real_time:16454.923826 global_time:44, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:10853
rax:0xffffffffffffffda rbx:0x22 rcx:0xffffffffffffffff rdx:0x3 rsi:0x2000 rdi:0x0 rbp:0x0 rsp:0x7ffdbdafd038 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x7ffdbdafd820 r13:0x24 r14:0x1 r15:0x0 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.923850 global_time:45, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:10853 rax:0x7f504fd51000 rbx:0x22 rcx:0xffffffffffffffff rdx:0x3 rsi:0x2000 rdi:0x0 rbp:0x0 rsp:0x7ffdbdafd038 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x7ffdbdafd820 r13:0x24 r14:0x1 r15:0x0 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"<ZERO>", addr:0x7f504fd51000, length:0x2000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x2000 } } { real_time:16454.923938 global_time:46, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:10966
rax:0xffffffffffffffda rbx:0x802 rcx:0xffffffffffffffff rdx:0x1 rsi:0x30360 rdi:0x0 rbp:0x0 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x0 r10:0x802 r11:0x246 r12:0x7f504fd51000 r13:0x1 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924045 global_time:47, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:10966 rax:0x7f504fd20000 rbx:0x802 rcx:0xffffffffffffffff rdx:0x1 rsi:0x30360 rdi:0x0 rbp:0x0 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x0 r10:0x802 r11:0x246 r12:0x7f504fd51000 r13:0x1 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"/usr/lib64/rr/librrpreload.so", addr:0x7f504fd20000, length:0x31000, prot_flags:"r--p", file_offset:0x0, device:66312, inode:5256379, data_file:"/usr/lib64/rr/librrpreload.so", data_offset:0x0, file_size:0x2a0b0 } } { real_time:16454.924076 global_time:48, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:10978
rax:0xffffffffffffffda rbx:0x812 rcx:0xffffffffffffffff rdx:0x5 rsi:0x25000 rdi:0x7f504fd21000 rbp:0x7f504fd21000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x1000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x5 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924114 global_time:49, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:10978 rax:0x7f504fd21000 rbx:0x812 rcx:0xffffffffffffffff rdx:0x5 rsi:0x25000 rdi:0x7f504fd21000 rbp:0x7f504fd21000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x1000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x5 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"/usr/lib64/rr/librrpreload.so", addr:0x7f504fd21000, length:0x25000, prot_flags:"r-xp", file_offset:0x1000, device:66312, inode:5256379, data_file:"/usr/lib64/rr/librrpreload.so", data_offset:0x1000, file_size:0x2a0b0 } } { real_time:16454.924145 global_time:50, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:10987
rax:0xffffffffffffffda rbx:0x812 rcx:0xffffffffffffffff rdx:0x1 rsi:0x3000 rdi:0x7f504fd46000 rbp:0x7f504fd46000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x26000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x1 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924182 global_time:51, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:10987 rax:0x7f504fd46000 rbx:0x812 rcx:0xffffffffffffffff rdx:0x1 rsi:0x3000 rdi:0x7f504fd46000 rbp:0x7f504fd46000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x26000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x1 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"/usr/lib64/rr/librrpreload.so", addr:0x7f504fd46000, length:0x3000, prot_flags:"r--p", file_offset:0x26000, device:66312, inode:5256379, data_file:"/usr/lib64/rr/librrpreload.so", data_offset:0x26000, file_size:0x2a0b0 } } { real_time:16454.924212 global_time:52, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:10996
rax:0xffffffffffffffda rbx:0x812 rcx:0xffffffffffffffff rdx:0x3 rsi:0x2000 rdi:0x7f504fd49000 rbp:0x7f504fd49000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x28000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x3 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924248 global_time:53, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:10996 rax:0x7f504fd49000 rbx:0x812 rcx:0xffffffffffffffff rdx:0x3 rsi:0x2000 rdi:0x7f504fd49000 rbp:0x7f504fd49000 rsp:0x7ffdbdafcf18 r8:0x3 r9:0x28000 r10:0x812 r11:0x246 r12:0x7f504fd51000 r13:0x3 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"/usr/lib64/rr/librrpreload.so", addr:0x7f504fd49000, length:0x2000, prot_flags:"rw-p", file_offset:0x28000, device:66312, inode:5256379, data_file:"/usr/lib64/rr/librrpreload.so", data_offset:0x28000, file_size:0x2a0b0 } } { real_time:16454.924281 global_time:54, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:31917, ticks:11068
rax:0xffffffffffffffda rbx:0x32 rcx:0xffffffffffffffff rdx:0x3 rsi:0x5360 rdi:0x7f504fd4b000 rbp:0x7f504fd4b000 rsp:0x7ffdbdafcf18 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7f504fd51000 r13:0x3 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924308 global_time:55, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:31917, ticks:11068 rax:0x7f504fd4b000 rbx:0x32 rcx:0xffffffffffffffff rdx:0x3 rsi:0x5360 rdi:0x7f504fd4b000 rbp:0x7f504fd4b000 rsp:0x7ffdbdafcf18 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7f504fd51000 r13:0x3 r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd71036 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0 { map_file:"<ZERO>", addr:0x7f504fd4b000, length:0x6000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x6000 } } { real_time:16454.924342 global_time:56, event:SYSCALL: close' (state:ENTERING_SYSCALL) tid:31917, ticks:11155
rax:0xffffffffffffffda rbx:0x6ffffdff rcx:0xffffffffffffffff rdx:0x7f504fd49f00 rsi:0x29 rdi:0x3 rbp:0x7ffdbdafd230 rsp:0x7ffdbdafd0d8 r8:0x7f504fd20000 r9:0xeffffef5 r10:0x70000022 r11:0x246 r12:0x7f504fd51000 r13:0x6ffffeff r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd70cdb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x0 gs_base:0x0
}
{
real_time:16454.924362 global_time:57, event:`SYSCALL: close' (state:EXITING_SYSCALL) tid:31917, ticks:11155
rax:0x0 rbx:0x6ffffdff rcx:0xffffffffffffffff rdx:0x7f504fd49f00 rsi:0x29 rdi:0x3 rbp:0x7ffdbdafd230 rsp:0x7ffdbdafd0d8 r8:0x7f504fd20000 r9:0xeffffef5 r10:0x70000022 r11:0x246 r12:0x7f504fd51000 r13:0x6ffffeff r14:0x7ffdbdafd308 r15:0x3 rip:0x7f504fd70cdb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x0 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x33)[0x8786d6]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x168)[0x70c350]
rr[0x730025]
rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x5a)[0x73020c]
rr(_ZN2rr9Registers22compare_register_filesEPNS_10ReplayTaskEPKcRKS0_S4_S6_NS_16MismatchBehaviorE+0x12a)[0x7da928]
rr(_ZN2rr10ReplayTask13validate_regsEj+0x2ee)[0x7f8e0c]
rr(_ZN2rr13ReplaySession13enter_syscallEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x73a)[0x7e389a]
rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0xd0)[0x7e708e]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x162)[0x7e8344]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xf1)[0x802b9b]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x571)[0x709b47]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x45a)[0x70b66e]
rr[0x7df2b3]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3b5)[0x7dfbd3]
rr(main+0x20c)[0x89010d]
/lib64/libc.so.6(__libc_start_main+0xf2)[0x7f114399f042]
rr(_start+0x2e)[0x67f62e]
=== End rr backtrace
Activity
rocallahan commentedon Mar 24, 2020
This suggests you updated the rr build between making the recording and performing the replay, and you didn't
rr packthe trace. Can you reproduce the problem without updating the rr build between recording and replay?stransky commentedon Mar 25, 2020
I tried record/replay with the custom build (and also with the rr rpm from the site) on firefox 74 / Fedora 32 and it fails with:
0x0000000070000002 in ?? ()
(rr) bt
#0 0x0000000070000002 in ?? ()
#1 0x00007ff5efc9dca2 in _raw_syscall () at /home/komat/tmp6/rr/src/preload/raw_syscall.S:120
#2 0x00007ff5efc9bda7 in traced_raw_syscall (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#3 syscall_hook_internal (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#4 syscall_hook (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2987
#5 0x00007ff5efc991ea in _syscall_hook_trampoline () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:282
#6 0x00007ff5efc9921a in __morestack () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:417
#7 0x00007ff5efc99220 in _syscall_hook_trampoline_48_3d_01_f0_ff_ff () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:423
#8 0x00007ff5efb3a3b1 in execve () from /lib64/libc.so.6
#9 0x000055d7f54cb126 in shell_execve ()
#10 0x000055d7f551f5f5 in exec_builtin ()
#11 0x000055d7f54c975e in execute_builtin ()
#12 0x000055d7f54ceb8c in execute_command_internal ()
#13 0x000055d7f54cf269 in execute_command ()
#14 0x000055d7f54b66a1 in reader_loop ()
#15 0x000055d7f54b5021 in main ()
rocallahan commentedon Mar 25, 2020
Can you run the rr tests? https://github.com/mozilla/rr/wiki/Building-And-Installing#tests
You'll want full output.
stransky commentedon Mar 25, 2020
There's the log here, Thanks.
test.txt
rocallahan commentedon Mar 25, 2020
Looks like a few issues here:
clock_adjtimeto implementadjtime, but newer systems don't. rr doesn't supportclock_adjtimeand unfortunately it's currently undocumented: https://lore.kernel.org/patchwork/patch/853181/clock_nanosleep_time64,clock_getres_time64andppoll_time64which is new post-2038 stuff I guess; we don't support any of those new syscalls yet. We'll need to fix this.daemon_32) that are probably related to your actual bug. Can yourr record .../daemon_32and thenrr replay -aand see what happens?rocallahan commentedon Mar 26, 2020
OK I think I addressed the first two issues: 53fa20b
I did not write tests for all the time64 syscalls. They would be redundant with existing tests as soon as test systems migrate to the libc F32 is using. So, you might find bugs in the code I wrote for some of those time64 syscalls. Shouldn't be hard to fix if so.
stransky commentedon Mar 26, 2020
Thanks. I tired to build it but fails with:
[komat@localhost obj]$ cmake ../rr
-- The C compiler identification is GNU 10.0.1
-- The CXX compiler identification is GNU 10.0.1
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/lib64/ccache/c++
-- Check for working CXX compiler: /usr/lib64/ccache/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORTS_CXX14
-- Performing Test SUPPORTS_CXX14 - Success
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.6.3")
-- Checking for module 'capnp'
-- Found capnp, version 0.7.0
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.2", minimum required is "3")
CMake Error: File /home/komat/tmp6/rr/src/test/clock_time64.c does not exist.
CMake Error at CMakeLists.txt:1420 (configure_file):
configure_file Problem configuring file
rocallahan commentedon Mar 26, 2020
Oops. Try again.
stransky commentedon Mar 26, 2020
hm, seems to be the same:
0x0000000070000002 in ?? ()
(rr) bt
#0 0x0000000070000002 in ?? ()
#1 0x00007f4addceaca2 in _raw_syscall () at /home/komat/tmp6/rr/src/preload/raw_syscall.S:120
#2 0x00007f4addce8da7 in traced_raw_syscall (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#3 syscall_hook_internal (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#4 syscall_hook (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2987
#5 0x00007f4addce61ea in _syscall_hook_trampoline () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:282
#6 0x00007f4addce621a in __morestack () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:417
#7 0x00007f4addce6220 in _syscall_hook_trampoline_48_3d_01_f0_ff_ff () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:423
#8 0x00007f4addb87461 in execve () from /lib64/libc.so.6
#9 0x00005573f232a126 in shell_execve ()
#10 0x00005573f237e5f5 in exec_builtin ()
#11 0x00005573f232875e in execute_builtin ()
#12 0x00005573f232db8c in execute_command_internal ()
#13 0x00005573f232e269 in execute_command ()
#14 0x00005573f23156a1 in reader_loop ()
#15 0x00005573f2314021 in main ()
rocallahan commentedon Mar 26, 2020
Right:
rocallahan commentedon Mar 26, 2020
There are some timeouts (e.g. in daemon_32) that are probably related to your actual bug. Can you rr record .../daemon_32 and then rr replay -a and see what happens?
stransky commentedon Mar 26, 2020
That looks ok:
[komat@localhost bin]$ rr record ./daemon_32
rr: Saving execution to trace directory `/home/komat/.local/share/rr/daemon_32-1'.
EXIT-SUCCESS
[komat@localhost bin]$ rr replay -a
EXIT-SUCCESS
rocallahan commentedon Mar 26, 2020
OK. Can you rerun the rr tests with the latest changes please?
ctest --verbose -j<...>would be helpful.stransky commentedon Mar 26, 2020
here we come:
run-tests.txt
rocallahan commentedon Mar 26, 2020
Some disturbing errors there. I'm not sure if any of them are related to your bug.
Can you
rr record constructor_32and thenrr replay -aand show the outputCan you show me exactly what you're doing to trigger this error replaying Firefox, and show me exactly what the error messages are?
3 remaining items
stransky commentedon Mar 27, 2020
there's a full output of the replay:
[komat@localhost firefox]$ rr replay
GNU gdb (GDB) Fedora 9.1-3.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/home/komat/.gdbinit:21: Error in sourced command file:
Ambiguous command "def prun": define, define-prefix.
Reading symbols from /home/komat/.local/share/rr/firefox-3/mmap_hardlink_3_firefox...
Missing separate debuginfo for /home/komat/.local/share/rr/firefox-3/mmap_hardlink_3_firefox
Try: dnf --enablerepo='debug' install /usr/lib/debug/.build-id/9c/333578720205cc5131ef8c080d04fe43ce81b3.debug
(No debugging symbols found in /home/komat/.local/share/rr/firefox-3/mmap_hardlink_3_firefox)
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:48827
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)
0x00007f9c0198a110 in _start () from /lib64/ld-linux-x86-64.so.2
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.31-2.fc32.x86_64
(rr) c
Continuing.
Program stopped.
0x0000000070000002 in ?? ()
Missing separate debuginfos, use: dnf debuginfo-install libgcc-10.0.1-0.9.fc32.x86_64 libstdc++-10.0.1-0.9.fc32.x86_64
(rr) bt
#0 0x0000000070000002 in ?? ()
#1 0x00007f9c0195aca2 in _raw_syscall () at /home/komat/tmp6/rr/src/preload/raw_syscall.S:120
#2 0x00007f9c01958da7 in traced_raw_syscall (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#3 syscall_hook_internal (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2953
#4 syscall_hook (call=0x681fffa0) at /home/komat/tmp6/rr/src/preload/syscallbuf.c:2987
#5 0x00007f9c019561ea in _syscall_hook_trampoline () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:282
#6 0x00007f9c0195621a in __morestack () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:417
#7 0x00007f9c01956220 in _syscall_hook_trampoline_48_3d_01_f0_ff_ff () at /home/komat/tmp6/rr/src/preload/syscall_hook.S:423
#8 0x00007f9c014b1461 in execve () from /lib64/libc.so.6
#9 0x00000000004011dc in ?? ()
#10 0x00007f9c0140a042 in __libc_start_main () from /lib64/libc.so.6
#11 0x00000000004010b9 in ?? ()
#12 0x00007ffdde837c78 in ?? ()
#13 0x00007f9c019b5fa0 in ?? () from /lib64/ld-linux-x86-64.so.2
#14 0x0000000000000001 in ?? ()
#15 0x00007ffdde8388f7 in ?? ()
#16 0x0000000000000000 in ?? ()
(rr)
rocallahan commentedon Mar 27, 2020
Ah! I thought there was an error or something. I think this is not a bug at all. rr doesn't support replaying with gdb through an execve. So, you need to work around this e.g.by using the -g option to select an rr event that's after the execve.
stransky commentedon Mar 30, 2020
Okay, I tried it and still does not work, replay of firefox record fails with:
Thread 5 received signal SIGKILL, Killed.
0x00007f0d93aca871 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(rr) bt
#0 0x00007f0d93aca871 in clock_nanosleep@GLIBC_2.2.5 () at /lib64/libc.so.6
#1 0x00007f0d93ad01c7 in nanosleep () at /lib64/libc.so.6
#2 0x00007f0d93afb929 in usleep () at /lib64/libc.so.6
#3 0x00007f0d8d329a6a in () at /home/komat/Programy/firefox/libxul.so
#4 0x00007f0d939dec44 in () at /home/komat/Programy/firefox/libnspr4.so
#5 0x00007f0d93f3a432 in start_thread () at /lib64/libpthread.so.0
#6 0x00007f0d93b039d3 in clone () at /lib64/libc.so.6
(rr)
rocallahan commentedon Mar 30, 2020
Can you please be clearer about what exactly you tried to do? There is no obvious error here. It's entirely possible that the process you attached to was killed by a SIGKILL.
rocallahan commentedon Mar 31, 2020
I should mention that rr reports
SIGKILLwhen the process exits for any reason, not just a realSIGKILL.stransky commentedon Mar 31, 2020
I still do the same, I just run rr on binaries downloaded from mozilla on Fedora 32, so I run:
$MOZ_ENABLE_WAYLAND=1 rr record ./firefox
which seems to work fine, firefox starts, quits, there's no error and then
$rr replay
which failed.
That works on Fedora 31 but fails on Fedora 32.
rocallahan commentedon Mar 31, 2020
As I said, the results in #2469 (comment) are normal behavior when you
continuein gdb and the recorded process exits. So there is no evidence of an rr bug there.I'm getting a bit frustrated. If you still think there is an rr bug, please provide a complete log of your debugging session and explain exactly why you think there is an rr bug.
stransky commentedon Mar 31, 2020
I tired rr with a simple code like this one:
https://developer.gnome.org/gtk3/stable/gtk-getting-started.html
I put breakpoint to exit() and it looks like you're right about the SIGKILL, exit() is executed first and then the SIGKILL. So it's correct although it's not very obvious to me that the program in rr finished correctly.
But there's another issue there. When I try to do reverse execution of the simple testcase (rn or rc), gdb freezes and prints:
(rr) rc
Continuing.
warning: Corrupted shared library list: 0xc25050 != 0xba20a0
of course I don't know if that's a bug in rr or gdb.
rocallahan commentedon Mar 31, 2020
It's a gdb issue really; gdb's handling of dynamic shared library loading doesn't work properly with reverse-execution. However, you can usually carry on without severe effects. Does gdb completely hang after that or do you eventually get back to a prompt? Reverse execution can be very slow if you go all the way back to the beginning of time.
stransky commentedon Mar 31, 2020
gdb really hangs on F32. I have two identical laptops (i7, 32gb ram, ssd), one has F31 and one F32.
both F31 and F32 prints:
warning: Corrupted shared library list: 0xc25050 != 0xba20a0
but F31 works as expected, reverse execution is performed, I've used rr a lot there while debugging various FF patches.
rocallahan commentedon Mar 31, 2020
Can you try building the gdb release in F31 on F32 and see if that avoids the hang? I'm pretty sure gdb hanging is a gdb issue.
stransky commentedon Mar 31, 2020
It needs some extra work as F32 uses gcc 10/gdb 9.1 and F31 gcc 9 / gdb 8. I tried to build gdb 8 on F32 but there are build errors there so I can't do that for now.
GitMensch commentedon Jul 16, 2021
No activity for over a year and before the last release. I suggest to close this issue, ideally with a no-activity marker.
rocallahan commentedon Jul 16, 2021
I agree.