Added PmuConfig for Intel Arrow Lake by jcpowermac · Pull Request #3906 · rr-debugger/rr

jcpowermac · 2025-01-24T19:17:15Z

This commit adds 0xc0660 as Intel Arrow Lake.

jcpowermac · 2025-01-24T19:17:51Z

very simple testing results

❯ taskset -c 0 ./bin/rr record echo "foo"
rr: Saving execution to trace directory `/home/jcallen/.local/share/rr/echo-1'.          
foo                                                                                      
❯ taskset -c 0 rr record echo "foo"                                                      
[FATAL src/PerfCounters_x86.h:154:compute_cpu_microarch()] Intel CPU type 0xc0660 unknown

khuey · 2025-01-24T21:00:18Z

Run make check please.

jcpowermac · 2025-01-27T13:35:06Z

Fails to complete unless taskset -c 0 make check is used. Tried just using make clean without taskset this caused many failures, assuming its being scheduled to the e-cores. Then changed to taskset -c 0-7 make check this also does not complete, let it sit over night and stops at 3125.

91% tests passed, 269 tests failed out of 3128                                                                                                                                                                                                 
                                                                                                                       
Total Test time (real) = 2654.67 sec                                                                                   
                                                                                                                       
The following tests FAILED:                                                                                            
         57 - chew_cpu (Failed)                 
         58 - chew_cpu-no-syscallbuf (Failed)                                                                          
        119 - detach_state (Failed)                                                                                                                                                                                                            
        120 - detach_state-no-syscallbuf (Failed)                                                                                                                                                                                              
        175 - x86/fault_in_code_page (Failed)     
        617 - x86/rdtsc (Failed)                                                                                                                                                                                                               
        619 - x86/rdtsc_flags (Failed)                  
        885 - unexpected_exit_execve (Failed)                                                                          
        889 - unexpected_exit_pid_ns (Failed)           
        890 - unexpected_exit_pid_ns-no-syscallbuf (Failed)                                                            
        975 - block_clone_interrupted (Failed)
        977 - block_clone_syscallbuf_overflow (Failed)                                                                 
        978 - block_clone_syscallbuf_overflow-no-syscallbuf (Failed)                                                                                                                                                                           
        985 - x86/blocked_sigsegv (Failed)                                                                                                                                                                                                     
        987 - breakpoint (Failed)                    
        988 - breakpoint-no-syscallbuf (Failed)                                                                                                                                                                                                
        1011 - clone_syscallbuf_cleanup_blocked (Failed)
        1021 - conditional_breakpoint_offload (Failed)                                                                 
        1022 - conditional_breakpoint_offload-no-syscallbuf (Failed)                                                   
        1029 - x86/cpuid_singlestep (Failed)                                                                           
        1030 - x86/cpuid_singlestep-no-syscallbuf (Failed)                                                             
        1047 - diversion_syscall (Failed)                                                                              
        1048 - diversion_syscall-no-syscallbuf (Failed)                                                                                                                                                                                        
        1049 - dlopen (Failed)                                                                                                                                                                                                                 
        1050 - dlopen-no-syscallbuf (Failed)                                                                           
        1073 - x86/explicit_checkpoints (Failed)                                                                                                                                                                                               
        1083 - x86/fxregs (Failed)                      
        1084 - x86/fxregs-no-syscallbuf (Failed)                                                                       
        1115 - jit_proc_mem (Failed)                                                                                   
        1116 - jit_proc_mem-no-syscallbuf (Failed)                                                                     
        1129 - mmap_replace_most_mappings (Failed)   
        1130 - mmap_replace_most_mappings-no-syscallbuf (Failed)          
        1143 - mprotect_checkpoint (Failed)                                                                                                                                                                                                    
        1144 - mprotect_checkpoint-no-syscallbuf (Failed)                                                                                                                                                                                      
        1145 - x86/morestack_unwind (Failed)                                                                           
        1171 - ptrace_remote_unmap (Failed)                                                                                                                                                                                                    
        1172 - ptrace_remote_unmap-no-syscallbuf (Failed)                                                              
        1173 - x86/rdtsc_loop (Failed)       
        1175 - x86/rdtsc_loop2 (Failed)                 
        1177 - x86/rdtsc_interfering (Failed)                                                                          
        1179 - read_big_struct (Failed)                                                                                
        1180 - read_big_struct-no-syscallbuf (Failed)                                                                  
        1189 - reverse_continue_process_signal (Failed)                                                                                                                                                                                        
        1190 - reverse_continue_process_signal-no-syscallbuf (Failed)                                                                                                                                                                          
        1207 - search (Failed)                    
        1208 - search-no-syscallbuf (Failed)                                                                                                                                                                                                   
        1235 - stack_growth (Failed)                                                                                   
        1236 - stack_growth-no-syscallbuf (Failed)                                                                                                                                                                                             
        1237 - step_into_lib (Failed)                                                                                                                                                                                                          
        1238 - step_into_lib-no-syscallbuf (Failed)                                                                    
        1240 - step_thread-no-syscallbuf (Failed)  
        1249 - x86/string_instructions_multiwatch (Failed)                                                             
        1250 - x86/string_instructions_multiwatch-no-syscallbuf (Failed)                                                                                                                                                                       
        1255 - x86/string_instructions_watch (Failed)                                                                                                                                                                                          
        1256 - x86/string_instructions_watch-no-syscallbuf (Failed)                                                    
        1261 - x86/syscallbuf_rdtsc_page (Failed)                                                                                                                                                                                              
        1269 - target_process (Failed)                                                                                 
        1270 - target_process-no-syscallbuf (Failed)
        1293 - tls (Failed)                                                                                            
        1294 - tls-no-syscallbuf (Failed)                                                                              
        1299 - unexpected_stack_growth (Failed)                                                                        
        1300 - unexpected_stack_growth-no-syscallbuf (Failed)       
        1301 - unicode (Failed)                                                                                                                                                                                                                
        1302 - unicode-no-syscallbuf (Failed)                                                                                                                                                                                                  
        1305 - vdso_clock_gettime_stack (Failed)  
        1307 - vdso_gettimeofday_stack (Failed)                                                                                                                                                                                                
        1309 - vdso_time_stack (Failed)                 
        1313 - vfork_read_clone_stress (Failed)                                                                        
        1314 - vfork_read_clone_stress-no-syscallbuf (Failed)                                                          
        1315 - vsyscall_reverse_next (Failed)                                                                          
        1316 - vsyscall_reverse_next-no-syscallbuf (Failed)                                                            
        1319 - watchpoint (Failed)                                                                                     
        1320 - watchpoint-no-syscallbuf (Failed)                                                                                                                                                                                               
        1321 - watchpoint_at_sched (Failed)                                                                                                                                                                                                    
        1322 - watchpoint_at_sched-no-syscallbuf (Failed)                                                              
        1323 - watchpoint_before_signal (Failed)                                                                                                                                                                                               
        1324 - watchpoint_before_signal-no-syscallbuf (Failed)                                                         
        1329 - watchpoint_size_change (Failed)                                                                         
        1330 - watchpoint_size_change-no-syscallbuf (Failed)                                                           
        1331 - watchpoint_syscall (Failed)                                                                             
        1332 - watchpoint_syscall-no-syscallbuf (Failed)
        1333 - watchpoint_unaligned (Failed)                                                                           
        1334 - watchpoint_unaligned-no-syscallbuf (Failed)                                                                                                                                                                                     
        1371 - call_exit (Failed)                                                                                                                                                                                                              
        1372 - call_exit-no-syscallbuf (Failed)   
        1375 - checkpoint_async_signal_syscalls_1000 (Failed)                                                                                                                                                                                  
        1381 - checkpoint_prctl_name (Failed)           
        1382 - checkpoint_prctl_name-no-syscallbuf (Failed)                                                            
        1383 - checkpoint_simple (Failed)               
        1384 - checkpoint_simple-no-syscallbuf (Failed)                                                                
        1399 - desched_ticks (Failed)                
        1407 - exec_stop (Failed)                                                                                      
        1408 - exec_stop-no-syscallbuf (Failed)                                                                                                                                                                                                
        1419 - fork_exec_info_thr (Failed)                                                                                                                                                                                                     
        1420 - fork_exec_info_thr-no-syscallbuf (Failed)
        1422 - get_thread_list-no-syscallbuf (Failed)                                                                                                                                                                                          
        1455 - remove_watchpoint (Failed)               
        1456 - remove_watchpoint-no-syscallbuf (Failed)                                                                
        1459 - replay_serve_files (Failed)              
        1460 - replay_serve_files-no-syscallbuf (Failed)                                                               
        1465 - restart_diversion (Failed)
        1466 - restart_diversion-no-syscallbuf (Failed)                                                                
        1469 - reverse_continue_exec_subprocess (Failed)                                                                                                                                                                                       
        1470 - reverse_continue_exec_subprocess-no-syscallbuf (Failed)                                                 
        1471 - reverse_continue_fork_subprocess (Failed)   
        1472 - reverse_continue_fork_subprocess-no-syscallbuf (Failed)                                                                                                                                                                         
        1473 - reverse_continue_int3 (Failed)                                                                          
        1474 - reverse_continue_int3-no-syscallbuf (Failed)                                                                                                                                                                                    
        1475 - reverse_continue_start (Failed)          
        1476 - reverse_continue_start-no-syscallbuf (Failed)                                                           
        1477 - reverse_finish (Failed)                  
        1478 - reverse_finish-no-syscallbuf (Failed)                                                                   
        1481 - reverse_step_signal (Failed)     
        1482 - reverse_step_signal-no-syscallbuf (Failed)                                                              
        1485 - reverse_watchpoint (Failed)                                                                                                                                                                                                     
        1486 - reverse_watchpoint-no-syscallbuf (Failed)                                                                                                                                                                                       
        1487 - reverse_watchpoint_syscall (Failed)       
        1488 - reverse_watchpoint_syscall-no-syscallbuf (Failed)                                                                                                                                                                               
        1509 - simple_script_debug (Failed)                                                                            
        1510 - simple_script_debug-no-syscallbuf (Failed)                                                              
        1511 - simple_winch (Failed)                                                                                   
        1512 - simple_winch-no-syscallbuf (Failed)                                                                     
        1513 - stack_overflow_debug (Failed)
        1514 - stack_overflow_debug-no-syscallbuf (Failed)                                                             
        1515 - step1 (Failed)                                                                                                                                                                                                                  
        1516 - step1-no-syscallbuf (Failed)                                                                                                                                                                                                    
        1517 - x86/step_rdtsc (Failed)                                                                
               1518 - x86/step_rdtsc-no-syscallbuf (Failed)
        1519 - step_signal (Failed)
        1520 - step_signal-no-syscallbuf (Failed)
        1521 - x86/string_instructions_break (Failed)
        1522 - x86/string_instructions_break-no-syscallbuf (Failed)
        1523 - x86/string_instructions_replay_quirk (Failed)
        1524 - x86/string_instructions_replay_quirk-no-syscallbuf (Failed)
        1547 - unwind_on_signal (Failed)
        1548 - unwind_on_signal-no-syscallbuf (Failed)
        1557 - watch_code (Failed)
        1558 - watch_code-no-syscallbuf (Failed)
        1561 - watchpoint_unaligned2 (Failed)
        1562 - watchpoint_unaligned2-no-syscallbuf (Failed)
        1563 - when (Failed)
        1564 - when-no-syscallbuf (Failed)
        1683 - detach_state-32 (Failed)
        1684 - detach_state-32-no-syscallbuf (Failed)
        2074 - pid_ns_shutdown-32-no-syscallbuf (Failed)
        2447 - unexpected_exit-32 (Failed)
        2453 - unexpected_exit_pid_ns-32 (Failed)
        2539 - block_clone_interrupted-32 (Failed)
        2551 - breakpoint-32 (Failed)
        2552 - breakpoint-32-no-syscallbuf (Failed)
        2575 - clone_syscallbuf_cleanup_blocked-32 (Failed)
        2585 - conditional_breakpoint_offload-32 (Failed)
        2586 - conditional_breakpoint_offload-32-no-syscallbuf (Failed)
        2593 - x86/cpuid_singlestep-32 (Failed)
        2594 - x86/cpuid_singlestep-32-no-syscallbuf (Failed)
        2597 - crash_in_function-32 (Failed)
        2598 - crash_in_function-32-no-syscallbuf (Failed)
        2611 - diversion_syscall-32 (Failed)
        2612 - diversion_syscall-32-no-syscallbuf (Failed)
        2613 - dlopen-32 (Failed)
        2614 - dlopen-32-no-syscallbuf (Failed)
        2647 - x86/fxregs-32 (Failed)
        2648 - x86/fxregs-32-no-syscallbuf (Failed)
        2677 - invalid_jump-32 (Failed)
        2678 - invalid_jump-32-no-syscallbuf (Failed)
        2679 - jit_proc_mem-32 (Failed)
        2680 - jit_proc_mem-32-no-syscallbuf (Failed)
        2693 - mmap_replace_most_mappings-32 (Failed)
        2694 - mmap_replace_most_mappings-32-no-syscallbuf (Failed)
        2707 - mprotect_checkpoint-32 (Failed)
        2708 - mprotect_checkpoint-32-no-syscallbuf (Failed)
        2709 - x86/morestack_unwind-32 (Failed)
        2735 - ptrace_remote_unmap-32 (Failed)
        2736 - ptrace_remote_unmap-32-no-syscallbuf (Failed)
        2743 - read_big_struct-32 (Failed)
        2744 - read_big_struct-32-no-syscallbuf (Failed)
        2753 - reverse_continue_process_signal-32 (Failed)
        2754 - reverse_continue_process_signal-32-no-syscallbuf (Failed)
        2771 - search-32 (Failed)
        2772 - search-32-no-syscallbuf (Failed)
        2779 - seekticks_threads-32 (Failed)
        2780 - seekticks_threads-32-no-syscallbuf (Failed)
        2799 - stack_growth-32 (Failed)
        2800 - stack_growth-32-no-syscallbuf (Failed)
        2801 - step_into_lib-32 (Failed)
        2802 - step_into_lib-32-no-syscallbuf (Failed)
        2813 - x86/string_instructions_multiwatch-32 (Failed)
        2814 - x86/string_instructions_multiwatch-32-no-syscallbuf (Failed)
        2819 - x86/string_instructions_watch-32 (Failed)
        2820 - x86/string_instructions_watch-32-no-syscallbuf (Failed)
        2833 - target_process-32 (Failed)
        2834 - target_process-32-no-syscallbuf (Failed)
        2857 - tls-32 (Failed)
        2858 - tls-32-no-syscallbuf (Failed)
        2863 - unexpected_stack_growth-32 (Failed)
        2864 - unexpected_stack_growth-32-no-syscallbuf (Failed)
        2865 - unicode-32 (Failed)
        2866 - unicode-32-no-syscallbuf (Failed)
        2877 - vfork_read_clone_stress-32 (Failed)
        2878 - vfork_read_clone_stress-32-no-syscallbuf (Failed)
        2879 - vsyscall_reverse_next-32 (Failed)
        2880 - vsyscall_reverse_next-32-no-syscallbuf (Failed)
        2883 - watchpoint-32 (Failed)
        2884 - watchpoint-32-no-syscallbuf (Failed)
        2885 - watchpoint_at_sched-32 (Failed)
        2886 - watchpoint_at_sched-32-no-syscallbuf (Failed)
        2887 - watchpoint_before_signal-32 (Failed)
        2888 - watchpoint_before_signal-32-no-syscallbuf (Failed)
        2893 - watchpoint_size_change-32 (Failed)
        2894 - watchpoint_size_change-32-no-syscallbuf (Failed)
        2895 - watchpoint_syscall-32 (Failed)
        2896 - watchpoint_syscall-32-no-syscallbuf (Failed)
        2897 - watchpoint_unaligned-32 (Failed)
        2898 - watchpoint_unaligned-32-no-syscallbuf (Failed)
        2935 - call_exit-32 (Failed)
        2936 - call_exit-32-no-syscallbuf (Failed)
        2963 - desched_ticks-32 (Failed)
        2971 - exec_stop-32 (Failed)
        2972 - exec_stop-32-no-syscallbuf (Failed)
        2983 - fork_exec_info_thr-32 (Failed)
        2984 - fork_exec_info_thr-32-no-syscallbuf (Failed)
        3019 - remove_watchpoint-32 (Failed)
        3020 - remove_watchpoint-32-no-syscallbuf (Failed)
        3023 - replay_serve_files-32 (Failed)
        3024 - replay_serve_files-32-no-syscallbuf (Failed)
        3029 - restart_diversion-32 (Failed)
        3030 - restart_diversion-32-no-syscallbuf (Failed)
        3033 - reverse_continue_exec_subprocess-32 (Failed)
        3034 - reverse_continue_exec_subprocess-32-no-syscallbuf (Failed)
        3035 - reverse_continue_fork_subprocess-32 (Failed)
        3036 - reverse_continue_fork_subprocess-32-no-syscallbuf (Failed)
        3037 - reverse_continue_int3-32 (Failed)
        3038 - reverse_continue_int3-32-no-syscallbuf (Failed)
        3039 - reverse_continue_start-32 (Failed)
        3040 - reverse_continue_start-32-no-syscallbuf (Failed)
        3041 - reverse_finish-32 (Failed)
        3042 - reverse_finish-32-no-syscallbuf (Failed)
        3043 - reverse_step_breakpoint-32 (Failed)
        3044 - reverse_step_breakpoint-32-no-syscallbuf (Failed)
        3045 - reverse_step_signal-32 (Failed)
        3046 - reverse_step_signal-32-no-syscallbuf (Failed)
        3049 - reverse_watchpoint-32 (Failed)
        3050 - reverse_watchpoint-32-no-syscallbuf (Failed)
        3051 - reverse_watchpoint_syscall-32 (Failed)
        3052 - reverse_watchpoint_syscall-32-no-syscallbuf (Failed)
        3073 - simple_script_debug-32 (Failed)
        3074 - simple_script_debug-32-no-syscallbuf (Failed)
        3075 - simple_winch-32 (Failed)
        3076 - simple_winch-32-no-syscallbuf (Failed)
        3077 - stack_overflow_debug-32 (Failed)
        3078 - stack_overflow_debug-32-no-syscallbuf (Failed)
        3079 - step1-32 (Failed)
        3080 - step1-32-no-syscallbuf (Failed)
        3081 - x86/step_rdtsc-32 (Failed)
        3082 - x86/step_rdtsc-32-no-syscallbuf (Failed)
        3083 - step_signal-32 (Failed)
        3084 - step_signal-32-no-syscallbuf (Failed)
        3085 - x86/string_instructions_break-32 (Failed)
        3086 - x86/string_instructions_break-32-no-syscallbuf (Failed)
        3087 - x86/string_instructions_replay_quirk-32 (Failed)
        3088 - x86/string_instructions_replay_quirk-32-no-syscallbuf (Failed)
        3111 - unwind_on_signal-32 (Failed)
        3112 - unwind_on_signal-32-no-syscallbuf (Failed)
        3121 - watch_code-32 (Failed)
        3122 - watch_code-32-no-syscallbuf (Failed)
        3125 - watchpoint_unaligned2-32 (Failed)
        3126 - watchpoint_unaligned2-32-no-syscallbuf (Failed)
        3127 - when-32 (Failed)
        3128 - when-32-no-syscallbuf (Failed)

khuey · 2025-01-27T17:16:01Z

Hmm, some of those test failures are concerning.

If you record and replay obj-rr/bin/rdtsc how does it fail?

jcpowermac · 2025-01-27T17:50:32Z

I really only use rr directly to record, then replay with delve e.g. dlv replay --backend=rr
so if there is specifics you need please let me know - I could be missing something obvious.

Running this on cachyos

Linux hydrogen 6.13.0-rc7-1-cachyos-rc #1 SMP PREEMPT_DYNAMIC Mon, 13 Jan 2025 17:14:57 +0000 x86_64 GNU/Linux

processor       : 0                                                                                                                                                                                          
vendor_id       : GenuineIntel                                                                                                                                                                               
cpu family      : 6                                                                                                                                                                                          
model           : 198                                                                                                                                                                                        
model name      : Intel(R) Core(TM) Ultra 7 265K                                                                                                                                                             
stepping        : 2                                                                                                                                                                                          
microcode       : 0x114                                                                                                                                                                                      
cpu MHz         : 800.000                                                                                                                                                                                    
cache size      : 30720 KB                                                                                                                                                                                   
physical id     : 0                                                                                                                                                                                          
siblings        : 20                                                                                                                                                                                         
core id         : 0                                                                                                                                                                                          
cpu cores       : 20                                                                                                                                                                                         
apicid          : 0                                                                                                                                                                                          
initial apicid  : 0                                                                                                                                                                                          
fpu             : yes                                                                                                                                                                                        
fpu_exception   : yes                                                                                                                                                                                        
cpuid level     : 35

 ❯ taskset -c 0 /bin/rr record ./bin/rdtsc
rr: Saving execution to trace directory `/home/jcallen/.local/share/rr/rdtsc-0'.
72025682973744,72025684533806,72025684677770,72025684807173,72025684939143,72025685066256,72025685195714,72025685320950,72025685448127,72025685571074,72025685695648,72025685819092,72025685942079,72025686065059,72025686191000,72025686315713,72025686439836,72025686564783,72025686689045,72025686812259,72025686935679,72025687059036,72025687180810,72025687305351,72025687428117,72025687576334,72025687700346,72025687824043,72025687948620,72025688072246,72025688194796,72025688320100,72025688444306,72025688568459,72025688691667,72025688814958,72025688937626,72025689062497,72025689185210,72025689308496,72025689430573,72025689552961,72025689677645,72025689801332,72025689924513,72025690046742,72025690169994,72025690294818,72025690420975,72025690545896,72025690669005,72025690792843,72025690918305,72025691042505,72025691165521,72025691288013,72025691411890,72025691542913,72025691666273,72025691790539,72025691913630,72025692038282,72025692160853,72025692285134,72025692412028,72025692536806,72025692661478,72025692784678,72025692908775,72025693031556,72025693157831,72025693284105,72025693407347,72025693530759,72025693655746,72025693780473,72025693905099,72025694029067,72025694154117,72025694278541,72025694402492,72025694526461,72025694653212,72025694775665,72025694901565,72025695026302,72025695149936,72025695273449,72025695406715,72025695531048,72025695655946,72025695779889,72025695903532,72025696028279,72025696155529,72025696277040,72025696401668,72025696527348,72025696651651,72025696775867,
EXIT-SUCCESS

❯ taskset -c 0 /bin/rr replay
GNU gdb (GDB) 16.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/jcallen/.local/share/rr/rdtsc-0/mmap_clone_5_rdtsc...
Remote debugging using 127.0.0.1:15657
Reading symbols from /lib64/ld-linux-x86-64.so.2...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
  <https://debuginfod.cachyos.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)
Failed to read a valid object file image from memory.
0x00007d6b8103fe40 in ?? () from /lib64/ld-linux-x86-64.so.2
(rr)

khuey · 2025-01-27T18:21:14Z

Sorry, please run rr replay -a. Should have been more specific.

jcpowermac · 2025-01-27T18:50:09Z

No worries... here you go.

❯ taskset -c 0 /bin/rr replay -a                                                                                                                                                                             
87377065419423,[FATAL src/ReplaySession.cc:1265:check_ticks_consistency()]                                                                                                                                   
 (task 535887 (rec:535730) at time 206)                                                                                                                                                                      
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: write'; expected 23535, got 23534
Tail of trace dump:
{                                                                                                                                                                                                            
  real_time:22506.079775 global_time:186, event:`SYSCALL: prlimit64' (state:ENTERING_SYSCALL) tid:535730, ticks:22369                                                                                        
rax:0xffffffffffffffda rbx:0x1 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x0 rbp:0x7ffc5c4bfc10 rsp:0x7ffc5c4bf8d8 r8:0x963cf85 r9:0x767902259000 r10:0x7ffc5c4bf8e0 r11:0x246 r12:0xfffffffffffffff8 r13:0x
767902291ab0 r14:0x0 r15:0x7679022922e0 rip:0x7679021159a4 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12e fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                                                                                                                            
  real_time:22506.079790 global_time:187, event:`SYSCALL: prlimit64' (state:EXITING_SYSCALL) tid:535730, ticks:22369                                                                                         
rax:0x0 rbx:0x1 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x0 rbp:0x7ffc5c4bfc10 rsp:0x7ffc5c4bf8d8 r8:0x963cf85 r9:0x767902259000 r10:0x7ffc5c4bf8e0 r11:0x246 r12:0xfffffffffffffff8 r13:0x767902291ab0 r1
4:0x0 r15:0x7679022922e0 rip:0x7679021159a4 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12e fs_base:0x767902210740 gs_base:0x0
  { tid:535730, addr:0x7ffc5c4bf8e0, length:0x10 }
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.079808 global_time:188, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:535730, ticks:22387               
rax:0xffffffffffffffda rbx:0x767902292128 rcx:0xffffffffffffffff rdx:0x0 rsi:0x224a7 rdi:0x767902218000 rbp:0x7ffc5c4bfc10 rsp:0x7ffc5c4bf8f8 r8:0x76790228fb78 r9:0x0 r10:0x7ffc5c4bf8e0 r11:0x246 r12:0xfff
ffffffffffff8 r13:0x767902291ab0 r14:0x0 r15:0x7679022922e0 rip:0x7679022809bb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x767902210740 gs_base:0x0
}                                                
{                                                
  real_time:22506.079826 global_time:189, event:`SYSCALL: munmap' (state:EXITING_SYSCALL) tid:535730, ticks:22387
rax:0x0 rbx:0x767902292128 rcx:0xffffffffffffffff rdx:0x0 rsi:0x224a7 rdi:0x767902218000 rbp:0x7ffc5c4bfc10 rsp:0x7ffc5c4bf8f8 r8:0x76790228fb78 r9:0x0 r10:0x7ffc5c4bf8e0 r11:0x246 r12:0xfffffffffffffff8 r
13:0x767902291ab0 r14:0x0 r15:0x7679022922e0 rip:0x7679022809bb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                     
  real_time:22506.079851 global_time:190, event:`SYSCALL: rrcall_init_preload' (state:ENTERING_SYSCALL) tid:535730, ticks:22610                                                                              
rax:0xffffffffffffffda rbx:0x7ffc5c4bf800 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x7ffc5c4bf7a0 rbp:0x1 rsp:0x7ffc5c4bf788 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffc5c4bfd28 r13:0x7ffc5c4bfd38 r14:0x76
7902247dd0 r15:0x0 rip:0x76790211eedd eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e8 fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                     
  real_time:22506.079898 global_time:191, event:`SYSCALL: rrcall_init_preload' (state:EXITING_SYSCALL) tid:535730, ticks:22610                                                                               
rax:0x0 rbx:0x7ffc5c4bf800 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x7ffc5c4bf7a0 rbp:0x1 rsp:0x7ffc5c4bf788 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffc5c4bfd28 r13:0x7ffc5c4bfd38 r14:0x767902247dd0 r15:
0x0 rip:0x76790211eedd eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e8 fs_base:0x767902210740 gs_base:0x0                                  
  { tid:535730, addr:0x7679022484a8, length:0x4000 }
  { tid:535730, addr:0x7679022484a2, length:0x1 }
  { tid:535730, addr:0x7679022484a3, length:0x1 }
  { tid:535730, addr:0x767902252268, length:0x8 }
  { tid:535730, addr:0x76790225227c, length:0x4 }                                                     
  { tid:535730, addr:0x767902252280, length:0x4 }                                                                                                                                                            
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.079923 global_time:192, event:`INSTRUCTION_TRAP' tid:535730, ticks:23219
rax:0xef2229f rbx:0x7ffc5c4bfd28 rcx:0x5f2491d1add8 rdx:0x4f78 rsi:0x7ffc5c4bfd28 rdi:0x1 rbp:0x7ffc5c4bfbd0 rsp:0x7ffc5c4bfbd0 r8:0x0 r9:0x76790225d500 r10:0x767902253000 r11:0x5 r12:0x0 r13:0x7ffc5c4bfd3
8 r14:0x5f2491d1add8 r15:0x767902291000 rip:0x5f2491d18385 eflags:0x10293 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                                                                                                                            
  real_time:22506.079971 global_time:193, event:`PATCH_SYSCALL' tid:535730, ticks:23219                                                                                                                      
rax:0xef2229f rbx:0x7ffc5c4bfd28 rcx:0x5f2491d1add8 rdx:0x4f78 rsi:0x7ffc5c4bfd28 rdi:0x1 rbp:0x7ffc5c4bfbd0 rsp:0x7ffc5c4bfbd0 r8:0x0 r9:0x76790225d500 r10:0x767902253000 r11:0x5 r12:0x0 r13:0x7ffc5c4bfd3
8 r14:0x5f2491d1add8 r15:0x767902291000 rip:0x5f2491d18385 eflags:0x10293 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x767902210740 gs_base:0x0
  { map_file:"<ZERO>", addr:0x5f2491d1d000, length:0x1000, prot_flags:"r-xp", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
  { tid:535730, addr:0x5f2491d1d000, length:0x63 }                                                                                                                                                           
  { tid:535730, addr:0x5f2491d18380, length:0x5 }                                                                                                                                                            
}                                                                                                                                                                                                            
{
  real_time:22506.080033 global_time:194, event:`PATCH_SYSCALL' tid:535730, ticks:23420
rax:0x1 rbx:0x7ffc5c4bfd28 rcx:0xffffffffffffffff rdx:0xf rsi:0x7ffc5c4bf710 rdi:0x1 rbp:0x7ffc5c4bfbd0 rsp:0x7ffc5c4bf6d8 r8:0xe r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x7ffc5c4bfd38 r14:0x5f2491d1add8 r15:
0x767902291000 rip:0x767902114c32 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x767902210740 gs_base:0x0                                                    
  { map_file:"<ZERO>", addr:0x767902202000, length:0x1000, prot_flags:"r-xp", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
  { tid:535730, addr:0x767902202000, length:0x5e }
  { tid:535730, addr:0x767902114c32, length:0x8 }
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.080051 global_time:195, event:`SYSCALL: gettid' (state:ENTERING_SYSCALL) tid:535730, ticks:23424                         
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x0 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x7ffc5c4bfd38 r14:0x681ffe30 r15:0x7679022910
00 rip:0x70000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xba fs_base:0x767902210740 gs_base:0x0
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.080062 global_time:196, event:`SYSCALL: gettid' (state:EXITING_SYSCALL) tid:535730, ticks:23424
rax:0x82cb2 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x0 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x7ffc5c4bfd38 r14:0x681ffe30 r15:0x767902291000 rip:0x70
000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xba fs_base:0x767902210740 gs_base:0x0
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.080079 global_time:197, event:`SYSCALL: perf_event_open' (state:ENTERING_SYSCALL) tid:535730, ticks:23424
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xffffffffffffffff rsi:0x0 rdi:0x681ffe30 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0xffffffffffffffff r11:0x246 r12:0x0 r13:0x7ffc5c4
bfd38 r14:0x681ffe30 r15:0x82cb2 rip:0x70000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12a fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                                                                                                                            
  real_time:22506.080101 global_time:198, event:`SYSCALL: perf_event_open' (state:EXITING_SYSCALL) tid:535730, ticks:23424                                                                                   
rax:0x3 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xffffffffffffffff rsi:0x0 rdi:0x681ffe30 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0xffffffffffffffff r11:0x246 r12:0x0 r13:0x7ffc5c4bfd38 r14:0x681
ffe30 r15:0x82cb2 rip:0x70000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12a fs_base:0x767902210740 gs_base:0x0
}
{                                                                                                                                                                                                            
  real_time:22506.080118 global_time:199, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:535730, ticks:23425                                                                                            
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x64 rsi:0x406 rdi:0x3 rbp:0x681ffef0 rsp:0x681ffd80 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip:0x700000
05 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x767902210740 gs_base:0x0                                                             
}                                                
{                                                
  real_time:22506.080133 global_time:200, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:535730, ticks:23425
rax:0x64 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x64 rsi:0x406 rdi:0x3 rbp:0x681ffef0 rsp:0x681ffd80 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip:0x70000005 eflags:0x24
6 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x767902210740 gs_base:0x0
  { tid:535730, addr:0x76790224850c, length:0x1 }                                                                                                                                                            
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.080151 global_time:201, event:`SYSCALL: rrcall_init_buffers' (state:ENTERING_SYSCALL) tid:535730, ticks:23430
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x681ffe30 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x64 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip:0x7
0000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e9 fs_base:0x767902210740 gs_base:0x0
}                                                                                                                                                                                                            
{                                                                                                     
  real_time:22506.080286 global_time:202, event:`SYSCALL: rrcall_init_buffers' (state:EXITING_SYSCALL) tid:535730, ticks:23430
rax:0x70002000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x681ffe30 rbp:0x681ffef0 rsp:0x681ffde0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x64 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip:0x70000005 
eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e9 fs_base:0x767902210740 gs_base:0x0
  { map_file:"<ZERO>", addr:0x70002000, length:0x100000, prot_flags:"rw-s", file_offset:0x0, device:41, inode:250449, data_file:"", data_offset:0x0, file_size:0x100000 }
  { tid:535730, addr:0x76790224850c, length:0x1 }
  { tid:535730, addr:0x7679022485d4, length:0x1 }
  { tid:535730, addr:0x681ffe30, length:0x20 }                                                                                                                                                               
}                                                                                                                                                                                                            
{                                                                                                                                                                                                            
  real_time:22506.080309 global_time:203, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:535730, ticks:23444
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xf rsi:0x7ffc5c4bf710 rdi:0x1 rbp:0x681ffef0 rsp:0x681ffdc0 r8:0xe r9:0x0 r10:0x0 r11:0x246 r12:0x64 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip
:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x767902210740 gs_base:0x0
}
{                      
  real_time:22506.080324 global_time:204, event:`SYSCALL: write' (state:EXITING_SYSCALL) tid:535730, ticks:23444
rax:0xf rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xf rsi:0x7ffc5c4bf710 rdi:0x1 rbp:0x681ffef0 rsp:0x681ffdc0 r8:0xe r9:0x0 r10:0x0 r11:0x246 r12:0x64 r13:0x3 r14:0x681ffe30 r15:0x82cb2 rip:0x70000002 efl
ags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x767902210740 gs_base:0x0
}                                                                                                     
{                                                                                                                                                                                                            
  real_time:22506.080345 global_time:205, event:`SYSCALLBUF_FLUSH' tid:535730, ticks:23535 
  { syscall:'rrcall_rdtsc', ret:0xf0a8061, size:0x18 }                            
}                                  
{                                  
  real_time:22506.080348 global_time:206, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:535730, ticks:23535                
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xf rsi:0x7ffc5c4bf710 rdi:0x1 rbp:0x681ffef0 rsp:0x681ffdc0 r8:0xe r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x7ffc5c4bfd38 r14:0x5f2491d1add8 r
15:0x767902291000 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x767902210740 gs_base:0x0
}                                                                                                     
{                                    
  real_time:22506.080352 global_time:207, event:`SYSCALLBUF_RESET' tid:535730, ticks:23535
}                   
=== Start rr backtrace:                                                                                                 
/bin/rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x5c) [0x5d611e4531f9]                                                                                                                                          
/bin/rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x116) [0x5d611e263a48]                                                                                                                                           
/bin/rr(+0x421ac6) [0x5d611e267ac6]                                                                   
/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x5f) [0x5d611e267d87]                                      
/bin/rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0x13f) [0x5d611e37dd87]                                                                                                 
/bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x8e6) [0x5d611e3829fc]
/bin/rr(_ZN2rr13ReplaySession11replay_stepENS_10RunCommandE+0x4b) [0x5d611e3762bd]
/bin/rr(+0x52ea12) [0x5d611e374a12]
/bin/rr(+0x52f06c) [0x5d611e37506c]
/bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x526) [0x5d611e37600c]
/bin/rr(main+0x283) [0x5d611e27a654]                                                                                                                                                                         
/usr/lib/libc.so.6(+0x261ce) [0x76b2dfe261ce]                                                                                                                                                                
/usr/lib/libc.so.6(__libc_start_main+0x8a) [0x76b2dfe2628a]                                           
/bin/rr(_start+0x25) [0x5d611e15a815]
=== End rr backtrace

khuey · 2025-01-27T19:59:28Z

Hmm, that's not good.

Are you running in any sort of VM?

jcpowermac · 2025-01-27T20:49:19Z

Hmm, that's not good.

Are you running in any sort of VM?

Nope, bare metal.

rocallahan · 2025-01-28T06:28:21Z

Maybe try the instructions in src/counters-test/README.md? with taskset 0.

jcpowermac · 2025-01-28T14:29:35Z

I had to modify counters to include arrow lake...

results performing for i in seq 1 1000; do taskset -c 0 /tmp/counters 1000 10000 > /tmp/output$i & done

❯ ls -1 output* | wc
    997     997    9862

❯ grep -i 'failed\|mismatch\|exit\|skid\|failed\|unable' output* | awk -F: '{print $2}' | uniq
EXIT-SUCCESS

not quite sure what the correct output should be but generally this is what the outfile files I checked

Interrupted after 10025 ticks, expected 10000 ticks
Interrupted after 10041 ticks, expected 10000 ticks
Interrupted after 10044 ticks, expected 10000 ticks
Interrupted after 10025 ticks, expected 10000 ticks
Interrupted after 10044 ticks, expected 10000 ticks
Interrupted after 10025 ticks, expected 10000 ticks
Interrupted after 10046 ticks, expected 10000 ticks
Interrupted after 10057 ticks, expected 10000 ticks
Interrupted after 10048 ticks, expected 10000 ticks
Interrupted after 10041 ticks, expected 10000 ticks
Interrupted after 10046 ticks, expected 10000 ticks
Interrupted after 10025 ticks, expected 10000 ticks

rocallahan · 2025-01-28T20:19:44Z

❯ ls -1 output* | wc
997 997 9862

So some files are missing? Why aren't there 1000 lines here?

If all output files have EXIT-SUCCESS then this didn't tell us anything...

jcpowermac · 2025-01-28T20:44:59Z

❯ ls -1 output* | wc
997 997 9862

So some files are missing? Why aren't there 1000 lines here?

zsh: job table full
title:16: job table full or recursion limit exceeded

If all output files have EXIT-SUCCESS then this didn't tell us anything...

rocallahan · 2025-01-28T20:56:05Z

When you rr replay -a the rdtsc recording, does it fail exactly the same way with the same numbers every single time?

If you rr pack the recording and put it somewhere we can download it for analysis, that might be helpful.

jcpowermac · 2025-01-28T21:20:26Z

running...
taskset -c 0 ./bin/rr record ./bin/rdtsc && taskset -c 0 ./bin/rr replay -a

result 1

=== Start rr backtrace:
./bin/rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x3d) [0x63982b4f1c5d]
./bin/rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x146) [0x63982b3d5ac6]
./bin/rr(+0xebaf9) [0x63982b3d8af9]
./bin/rr(+0xed59f) [0x63982b3da59f]
./bin/rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0x19c) [0x63982b46a81c]
./bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x3b4) [0x63982b471d94]
./bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x16a2) [0x63982b464f32]
./bin/rr(main+0x1a0) [0x63982b346c30]
/usr/lib/libc.so.6(+0x261ce) [0x738db3c261ce]
/usr/lib/libc.so.6(__libc_start_main+0x8a) [0x738db3c2628a]
./bin/rr(_start+0x25) [0x63982b348535]
=== End rr backtrace

result 2

=== Start rr backtrace:
./bin/rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x3d) [0x5559a3812c5d]
./bin/rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x146) [0x5559a36f6ac6]
./bin/rr(+0xebaf9) [0x5559a36f9af9]
./bin/rr(+0xed59f) [0x5559a36fb59f]
./bin/rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0x19c) [0x5559a378b81c]
./bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x3b4) [0x5559a3792d94]
./bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x16a2) [0x5559a3785f32]
./bin/rr(main+0x1a0) [0x5559a3667c30]
/usr/lib/libc.so.6(+0x261ce) [0x7cc6930261ce]
/usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7cc69302628a]
./bin/rr(_start+0x25) [0x5559a3669535]
=== End rr backtrace

re-running to get output of multiple runs

taskset -c 0 ./bin/rr record ./bin/rdtsc && taskset -c 0 ./bin/rr replay -a &> /tmp/runone.txt                                                                                                        
taskset -c 0 ./bin/rr record ./bin/rdtsc && taskset -c 0 ./bin/rr replay -a &> /tmp/runtwo.txt

runtwo.txt
runone.txt

@rocallahan you should have received an email from me with the link to google drive.

rocallahan · 2025-01-29T20:20:46Z

@rocallahan you should have received an email from me with the link to google drive.

Yes, thank you, this is very helpful.

Was this recorded while running in a VM? If so, which hypervisor?

jcpowermac · 2025-01-29T20:28:34Z

@rocallahan you should have received an email from me with the link to google drive.

Yes, thank you, this is very helpful.

Was this recorded while running in a VM? If so, which hypervisor?

Nope, bare metal

I am using CachyOS with their kernel features, currently

> uname -a
Linux hydrogen 6.13.0-rc7-1-cachyos-rc

https://wiki.cachyos.org/features/kernel/
https://github.com/firelzrd/bore-scheduler

khuey · 2025-01-29T20:42:36Z

Could you run rr rerun --singlestep=event,ticks,rip --trace-end 17 on the trace you shared with @rocallahan and pastebin the output somewhere?

jcpowermac · 2025-01-29T20:49:20Z

Could you run rr rerun --singlestep=event,ticks,rip --trace-end 17 on the trace you shared with @rocallahan and pastebin the output somewhere?

taskset -c 0 ./bin/rr rerun --singlestep=event,ticks,rip --trace-end 17 /home/jcallen/.local/share/rr/rdtsc-14 > rerun.txt

https://gist.github.com/jcpowermac/bd79079d6c2c5e4b1b9043b4f0374b7b

khuey · 2025-01-29T21:08:35Z

Thanks. Unfortunately those results are not good. For example,

0x70d545c70bc9: jnz $0x000070d545c70be7

does not register as a tick (i.e. a retired conditional branch) on your system but it does on ours.

There's no obvious pattern either.

0x70d545c70beb: jbe $0x000070d545c70bd6 does register as a tick, but

0x70d545c70c24: jbe $0x000070d545c70d10 does not.

khuey · 2025-01-29T21:09:41Z

We may need to have a backchannel discussion with $CHIP_VENDOR here.

khuey · 2025-01-29T21:16:32Z

Actually maybe there is a pattern.

0x70d545c70be5: jz $0x000070d545c70c38 appears to be part of a loop and is counted as a tick every time it's not taken, and then in the final appearance in the output where it is taken it is not counted as a tick. The counter may be counting correctly predicted branches rather than all retired branches. That would also explain how it gives different results on the same machine, rather than just different results on different machines.

rocallahan · 2025-01-29T22:23:08Z

Do we know for sure that core 0 is a P-Core and not an E-Core on this system?

jcpowermac · 2025-01-29T22:50:10Z

Do we know for sure that core 0 is a P-Core and not an E-Core on this system?

❯ cat /sys/devices/cpu_core/cpus
0-7
❯ cat /sys/devices/cpu_atom/cpus
8-19

❯ lscpu --all --extended
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 5100.0000 800.0000  739.3040
  1    0      0    1 4:4:1:0          yes 5000.0000 800.0000  800.0000
  2    0      0    2 16:16:4:0        yes 5000.0000 800.0000  780.8030
  3    0      0    3 20:20:5:0        yes 5000.0000 800.0000  800.0000
  4    0      0    4 24:24:6:0        yes 5000.0000 800.0000  800.0000
  5    0      0    5 28:28:7:0        yes 5000.0000 800.0000  800.0000
  6    0      0    6 40:40:10:0       yes 5000.0000 800.0000  800.0000
  7    0      0    7 44:44:11:0       yes 5100.0000 800.0000  740.3550
  8    0      0    8 3:0              yes 4600.0000 800.0000 4561.1211
  9    0      0    9 26:26:3:0        yes 4600.0000 800.0000 4532.8418
 10    0      0   10 3:0              yes 4600.0000 800.0000  800.0000
 11    0      0   11 30:30:3:0        yes 4600.0000 800.0000  834.1410
 12    0      0   12 64:64:8:0        yes 4600.0000 800.0000  800.0000
 13    0      0   13 66:66:8:0        yes 4600.0000 800.0000  558.9650
 14    0      0   14 68:68:8:0        yes 4600.0000 800.0000  800.0000
 15    0      0   15 70:70:8:0        yes 4600.0000 800.0000  800.0000
 16    0      0   16 72:72:9:0        yes 4600.0000 800.0000  731.7380
 17    0      0   17 74:74:9:0        yes 4600.0000 800.0000  800.0000
 18    0      0   18 76:76:9:0        yes 4600.0000 800.0000 4613.5542
 19    0      0   19 78:78:9:0        yes 4600.0000 800.0000  866.1110

https://www.intel.com/content/www/us/en/products/sku/241063/intel-core-ultra-7-processor-265k-30m-cache-up-to-5-50-ghz/specifications.html

https://stackoverflow.com/questions/71122837/how-to-detect-e-cores-and-p-cores-in-linux-alder-lake-system#71282744

rocallahan · 2025-01-30T00:49:55Z

Can you forward us the output of

cat /sys/devices/cpu_core/format/event

and

cat /sys/devices/cpu_core/format/umask

? Thanks!!!

rocallahan · 2025-01-30T10:13:38Z

The counter may be counting correctly predicted branches rather than all retired branches.

I suspect it's not as simple as that... if it was, then I think counters.cc would fail at least sometimes because we'd mispredict some of the conditional branches in test_ticks_basic().

jcpowermac · 2025-01-30T14:45:33Z

❯ cat /sys/devices/cpu_core/format/event
config:0-7
❯ cat /sys/devices/cpu_core/format/umask
config:8-15,40-47

khuey · 2025-01-30T17:13:24Z

I suspect it's not as simple as that

Sure, I was just speculating.

khuey · 2025-01-31T01:42:25Z

@jcpowermac, could you try using 0x100005111c4 instead of 0x5111c4?

jcpowermac · 2025-02-05T18:13:55Z

Sorry for the slow response... I got tired of dealing w/cachyos (arch) and went back to fedora (uname -a - Linux fedora 6.13.0-361.vanilla.fc41.x86_64)

There are still failures but not as many

taskset -c 0 make check     
...

3128/3128 Test #3099: trace_version-32 ..................................................   Passed  131.48 sec

99% tests passed, 32 tests failed out of 3128

Total Test time (real) = 1166.42 sec

The following tests FAILED:
        119 - detach_state (Failed)
        120 - detach_state-no-syscallbuf (Failed)
        175 - x86/fault_in_code_page (Failed)
        617 - x86/rdtsc (Failed)
        619 - x86/rdtsc_flags (Failed)
        883 - unexpected_exit (Failed)
        884 - unexpected_exit-no-syscallbuf (Failed)
        885 - unexpected_exit_execve (Failed)
        887 - unexpected_exit_execve_twice (Failed)
        888 - unexpected_exit_execve_twice-no-syscallbuf (Failed)
        889 - unexpected_exit_pid_ns (Failed)
        890 - unexpected_exit_pid_ns-no-syscallbuf (Failed)
        985 - x86/blocked_sigsegv (Failed)
        1073 - x86/explicit_checkpoints (Failed)
        1173 - x86/rdtsc_loop (Failed)
        1175 - x86/rdtsc_loop2 (Failed)
        1177 - x86/rdtsc_interfering (Failed)
        1261 - x86/syscallbuf_rdtsc_page (Failed)
        1323 - watchpoint_before_signal (Failed)
        1513 - stack_overflow_debug (Failed)
        1514 - stack_overflow_debug-no-syscallbuf (Failed)
        1683 - detach_state-32 (Failed)
        1684 - detach_state-32-no-syscallbuf (Failed)
        2447 - unexpected_exit-32 (Failed)
        2448 - unexpected_exit-32-no-syscallbuf (Failed)
        2449 - unexpected_exit_execve-32 (Failed)
        2450 - unexpected_exit_execve-32-no-syscallbuf (Failed)
        2451 - unexpected_exit_execve_twice-32 (Failed)
        2453 - unexpected_exit_pid_ns-32 (Failed)
        2454 - unexpected_exit_pid_ns-32-no-syscallbuf (Failed)
        3077 - stack_overflow_debug-32 (Failed)
        3078 - stack_overflow_debug-32-no-syscallbuf (Failed)
Errors while running CTest
Output from these tests are in: /home/jcallen/Development/rr/obj/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make[3]: *** [CMakeFiles/check.dir/build.make:70: CMakeFiles/check] Error 8
make[2]: *** [CMakeFiles/Makefile2:38503: CMakeFiles/check.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:38510: CMakeFiles/check.dir/rule] Error 2
make: *** [Makefile:18035: check] Error 2

jcpowermac · 2025-02-05T18:17:39Z

Still getting with...

  453  taskset -c 0 ./bin/rr record ./bin/rdtsc
  454  taskset -c 0 ./bin/rr replay -a

➜  obj git:(add-arrow-lake) ✗ taskset -c 0 ./bin/rr record ./bin/rdtsc
rr: Saving execution to trace directory `/home/jcallen/.local/share/rr/rdtsc-1'.
1638736433800005,1638736434836958,1638736434912690,1638736434982661,1638736435051460,1638736435119227,1638736435185040,1638736435253722,1638736435321703,1638736435387673,1638736435454884,1638736435518167,1638736435584514,1638736435647744,1638736435711111,1638736435774880,1638736435838092,1638736435900670,1638736435964657,1638736436027891,1638736436094422,1638736436158454,1638736436222870,1638736436290627,1638736436353822,1638736436417119,1638736436480870,1638736436544876,1638736436608231,1638736436673722,1638736436738037,1638736436802798,1638736436865596,1638736436929090,1638736436991660,1638736437057914,1638736437120629,1638736437183588,1638736437246909,1638736437313975,1638736437378469,1638736437442087,1638736437505227,1638736437568396,1638736437631523,1638736437694602,1638736437759641,1638736437822407,1638736437886807,1638736437951606,1638736438017420,1638736438081409,1638736438152515,1638736438218848,1638736438283462,1638736438347821,1638736438411176,1638736438473701,1638736438541537,1638736438610049,1638736438673792,1638736438737299,1638736438800509,1638736438864036,1638736438928383,1638736438992268,1638736439056144,1638736439119863,1638736439183107,1638736439249965,1638736439312840,1638736439375827,1638736439441221,1638736439505586,1638736439569832,1638736439633868,1638736439699078,1638736439765112,1638736439831632,1638736439896123,1638736439959749,1638736440025938,1638736440089368,1638736440153426,1638736440216909,1638736440281016,1638736440349336,1638736440414905,1638736440478749,1638736440542157,1638736440606080,1638736440669581,1638736440735152,1638736440798731,1638736440863101,1638736440929813,1638736440995122,1638736441060271,1638736441123933,1638736441188177,
EXIT-SUCCESS

=== Start rr backtrace:
./bin/rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x2a) [0x5ba3ca]
./bin/rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x118) [0x4a1de8]
./bin/rr() [0x4a4bf6]
./bin/rr() [0x4a64bf]
./bin/rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0x174) [0x52f714]
./bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x3ca) [0x53649a]
./bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x1647) [0x52a277]
./bin/rr(main+0x190) [0x41d460]
/lib64/libc.so.6(+0x3248) [0x7f28c052a248]
/lib64/libc.so.6(__libc_start_main+0x8b) [0x7f28c052a30b]
./bin/rr(_start+0x25) [0x41eaf5]
=== End rr backtrace

jcpowermac · 2025-02-05T18:20:57Z

@jcpowermac, could you try using 0x100005111c4 instead of 0x5111c4?

/home/jcallen/Development/rr/src/PerfCounters.cc:206:40: error: narrowing conversion of ‘1099516940740’ from ‘long int’ to ‘unsigned int’ [-Wnarrowing]
  206 |   { IntelArrowLake, "Intel Arrowlake", 0x100005111c4, 0, 0, 125, PMU_TICKS_RCB },
      |                                        ^~~~~~~~~~~~~
make[2]: *** [CMakeFiles/rr.dir/build.make:699: CMakeFiles/rr.dir/src/PerfCounters.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2957: CMakeFiles/rr.dir/all] Error 2

jcpowermac · 2025-02-05T18:26:07Z

Changed

-  unsigned rcb_cntr_event;
+  unsigned long rcb_cntr_event;

jcpowermac · 2025-02-05T18:59:10Z

One less test failed with 0x100005111c4


Total Test time (real) = 1218.59 sec

The following tests FAILED:
        119 - detach_state (Failed)
        120 - detach_state-no-syscallbuf (Failed)
        175 - x86/fault_in_code_page (Failed)
        617 - x86/rdtsc (Failed)
        619 - x86/rdtsc_flags (Failed)
        885 - unexpected_exit (Failed)
        887 - unexpected_exit_execve (Failed)
        889 - unexpected_exit_execve_twice (Failed)
        890 - unexpected_exit_execve_twice-no-syscallbuf (Failed)
        891 - unexpected_exit_pid_ns (Failed)
        892 - unexpected_exit_pid_ns-no-syscallbuf (Failed)
        987 - x86/blocked_sigsegv (Failed)
        1075 - x86/explicit_checkpoints (Failed)
        1175 - x86/rdtsc_loop (Failed)
        1177 - x86/rdtsc_loop2 (Failed)
        1179 - x86/rdtsc_interfering (Failed)
        1263 - x86/syscallbuf_rdtsc_page (Failed)
        1325 - watchpoint_before_signal (Failed)
        1455 - record_replay (Failed)
        1515 - stack_overflow_debug (Failed)
        1516 - stack_overflow_debug-no-syscallbuf (Failed)
        1685 - detach_state-32 (Failed)
        1686 - detach_state-32-no-syscallbuf (Failed)
        2451 - unexpected_exit-32 (Failed)
        2453 - unexpected_exit_execve-32 (Failed)
        2454 - unexpected_exit_execve-32-no-syscallbuf (Failed)
        2455 - unexpected_exit_execve_twice-32 (Failed)
        2457 - unexpected_exit_pid_ns-32 (Failed)
        2458 - unexpected_exit_pid_ns-32-no-syscallbuf (Failed)
        3081 - stack_overflow_debug-32 (Failed)
        3082 - stack_overflow_debug-32-no-syscallbuf (Failed)
        ```

khuey · 2025-02-05T19:14:30Z

If the rdtsc test is still failing I don't think that helped.

rocallahan · 2025-02-05T20:54:26Z

I think more code changes are needed to use the extended umask. In particular init_perf_event_attr()'s config parameter is unsigned so I think the extended umask was lost in the call to init_perf_event_attr().

rocallahan · 2025-02-05T21:02:32Z

OK, commit d7a8309 fixes that issue. Please retry with 0x100005111c4 as the event config value.

This commit adds `0xc0660` as Intel Arrow Lake.

jcpowermac · 2025-02-05T22:09:37Z

Thanks @rocallahan @khuey !
down to four failures

99% tests passed, 4 tests failed out of 3132

Total Test time (real) = 1008.56 sec

The following tests FAILED:
        119 - detach_state (Failed)
        120 - detach_state-no-syscallbuf (Failed)
        1685 - detach_state-32 (Failed)
        1686 - detach_state-32-no-syscallbuf (Failed)
Errors while running CTest

khuey · 2025-02-05T22:16:01Z

\o/

rocallahan · 2025-02-05T22:24:04Z

That remaining test failure is not hardware dependent. Thanks for all the help!

Added PmuConfig for Intel Arrow Lake

116acdf

This commit adds `0xc0660` as Intel Arrow Lake.

jcpowermac force-pushed the add-arrow-lake branch from 07e00fb to 116acdf Compare February 5, 2025 22:09

rocallahan merged commit 766e0e8 into rr-debugger:master Feb 5, 2025

Conversation

jcpowermac commented Jan 24, 2025

Uh oh!

jcpowermac commented Jan 24, 2025

Uh oh!

khuey commented Jan 24, 2025

Uh oh!

jcpowermac commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khuey commented Jan 27, 2025

Uh oh!

jcpowermac commented Jan 27, 2025

Uh oh!

khuey commented Jan 27, 2025

Uh oh!

jcpowermac commented Jan 27, 2025

Uh oh!

khuey commented Jan 27, 2025

Uh oh!

jcpowermac commented Jan 27, 2025

Uh oh!

rocallahan commented Jan 28, 2025

Uh oh!

jcpowermac commented Jan 28, 2025

Uh oh!

rocallahan commented Jan 28, 2025

Uh oh!

jcpowermac commented Jan 28, 2025

Uh oh!

rocallahan commented Jan 28, 2025

Uh oh!

jcpowermac commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocallahan commented Jan 29, 2025

Uh oh!

jcpowermac commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khuey commented Jan 29, 2025

Uh oh!

jcpowermac commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khuey commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khuey commented Jan 29, 2025

Uh oh!

khuey commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocallahan commented Jan 29, 2025

Uh oh!

jcpowermac commented Jan 29, 2025

Uh oh!

rocallahan commented Jan 30, 2025

Uh oh!

rocallahan commented Jan 30, 2025

Uh oh!

jcpowermac commented Jan 30, 2025

Uh oh!

khuey commented Jan 30, 2025

Uh oh!

khuey commented Jan 31, 2025

Uh oh!

jcpowermac commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcpowermac commented Feb 5, 2025

Uh oh!

jcpowermac commented Feb 5, 2025

Uh oh!

jcpowermac commented Feb 5, 2025

Uh oh!

jcpowermac commented Feb 5, 2025

Uh oh!

khuey commented Feb 5, 2025

Uh oh!

jcpowermac commented Jan 27, 2025 •

edited

Loading

jcpowermac commented Jan 28, 2025 •

edited

Loading

jcpowermac commented Jan 29, 2025 •

edited

Loading

jcpowermac commented Jan 29, 2025 •

edited

Loading

khuey commented Jan 29, 2025 •

edited

Loading

khuey commented Jan 29, 2025 •

edited

Loading

jcpowermac commented Feb 5, 2025 •

edited

Loading