rr record & reply fails on windriver linux

Below 3 issues seen with Windriver Linux.

$ rr record ./test1_64
rr: Saving execution to trace directory `/data/var/home/root/.local/share/rr/test1_64-0'.
[FATAL /b/syrajendra/git-rr/rr/src/Task.cc:2683:spawn() errno: ENOENT] Error checking fd
. . .
Aborted (core dumped)

$ rr replay /data/var/home/root/.local/share/rr/test1_64-1
[FATAL /b/syrajendra/git-rr/rr/src/Task.cc:2683:spawn() errno: EINVAL] Error checking fd

$ rr replay /data/var/home/root/.local/share/rr/test1_64-1
[FATAL /b/syrajendra/git-rr/rr/src/util.cc:1296:open_socket() errno: EADDRNOTAVAIL] Couldn't 
bind to port 23079

Below change fixes above problems:

diff --git a/src/Task.cc b/src/Task.cc
index 20005db..4f10b3f 100644
--- a/src/Task.cc
+++ b/src/Task.cc
@@ -2679,7 +2679,7 @@ static void run_initial_child(Session& session, const ScopedFd& 
error_fd,
   while (true) {
     ret = fcntl(fd_number, F_GETFD);
     if (ret < 0) {
-      if (errno != EBADF) {
+      if (errno != EBADF && errno != ENOENT && errno != EINVAL) {
         FATAL() << "Error checking fd";
       }
       break;

diff --git a/src/util.cc b/src/util.cc
index 2edae23..f4d6efd 100644
--- a/src/util.cc
+++ b/src/util.cc
@@ -1289,7 +1289,7 @@ ScopedFd open_socket(const char* address, unsigned short* port,
   do {
     addr.sin_port = htons(*port);
     ret = ::bind(listen_fd, (struct sockaddr*)&addr, sizeof(addr));
-    if (ret && (EADDRINUSE == errno || EACCES == errno || EINVAL == errno)) {
+    if (ret && (EADDRINUSE == errno || EACCES == errno || EINVAL == errno || 
EADDRNOTAVAIL == errno)) {
       continue;
     }
     if (ret) {

I'd like to understand the root cause. Do you get this on all testcases or just your test1_64 testcase?

If you run rr record under gdb using gdb --args rr record ..., then when you hit the error, can you check the fd and this->tid in Task::spawn, and look in /proc/.../fd/... to see what that fd actually is?

Here is the backtrace & other values
(gdb) bt
#0 __GI_abort () at /usr/src/debug/glibc/2.24-r0/git/stdlib/abort.c:51
#1 0x0000000000875369 in rr::notifying_abort () at /b/syrajendra/git-rr/rr/src/util.cc:1325
#2 0x000000000074398a in rr::FatalOstream::~FatalOstream (this=0x7fffffffe1ea, __in_chrg=)
at /b/syrajendra/git-rr/rr/src/log.cc:360
#3 0x000000000084a438 in rr::Task::spawn (session=..., error_fd=..., sock_fd_out=0xcbfdf0, tracee_socket_fd_number_out=0xcbfaf8,
trace=..., exe_path=..., argv=..., envp=..., rec_tid=-1) at /b/syrajendra/git-rr/rr/src/Task.cc:2683
#4 0x0000000000781edf in rr::RecordSession::RecordSession (this=0xcbfa30, exe_path=..., argv=..., envp=...,
disable_cpuid_features=..., syscallbuf=rr::RecordSession::ENABLE_SYSCALL_BUF, syscallbuf_desched_sig=30, bind_cpu=rr::BIND_CPU,
output_trace_dir=..., trace_id=0x0) at /b/syrajendra/git-rr/rr/src/RecordSession.cc:1952
#5 0x00000000007819ce in rr::RecordSession::create (argv=..., extra_env=..., disable_cpuid_features=...,
syscallbuf=rr::RecordSession::ENABLE_SYSCALL_BUF, syscallbuf_desched_sig=30 '\036', bind_cpu=rr::BIND_CPU, output_trace_dir=...,
trace_id=0x0) at /b/syrajendra/git-rr/rr/src/RecordSession.cc:1913
#6 0x0000000000776b84 in rr::record (args=..., flags=...) at /b/syrajendra/git-rr/rr/src/RecordCommand.cc:582
#7 0x00000000007776b8 in rr::RecordCommand::run (this=0xca9fd0 rr::RecordCommand::singleton, args=...)
at /b/syrajendra/git-rr/rr/src/RecordCommand.cc:729
#8 0x000000000088bc9d in main (argc=3, argv=0x7fffffffeb28) at /b/syrajendra/git-rr/rr/src/main.cc:268
(gdb) f 3
#3 0x000000000084a438 in rr::Task::spawn (session=..., error_fd=..., sock_fd_out=0xcbfdf0, tracee_socket_fd_number_out=0xcbfaf8,
trace=..., exe_path=..., argv=..., envp=..., rec_tid=-1) at /b/syrajendra/git-rr/rr/src/Task.cc:2683
2683 /b/syrajendra/git-rr/rr/src/Task.cc: No such file or directory.
(gdb) p fd_number
$1 = 1001
(gdb) p rec_tid
$2 = -1
(gdb) p tid
$3 = 0
(gdb) info proc
process 23428
warning: target file /proc/23428/cmdline contained unexpected null characters
cmdline = '/var/tmp/rr/bin/rr'
cwd = '/var/tmp'
exe = '/var/tmp/rr/bin/rr'
(gdb) shell ls /proc/23428/fd/
0 1 10 11 2 3 4 5 6 7 8 9

Can you compile and run this C program and tell me what you get?

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(void) {
  int i;
  for (i = 0; i < 2048; ++i) {
    errno = 0;
    int ret = fcntl(i, F_GETFD);
    printf("Fd %d: %d, err=%s\n", i, ret, strerror(errno));
  }
  return 0;
}

Here is the output of the program
$ ./test2_64
Fd 0: 0, err=Success
Fd 1: 0, err=Success
Fd 2: 0, err=Success
Fd 3: -1, err=Success
Fd 4: -1, err=Success
Fd 5: -1, err=Success
. . .
Fd 2045: -1, err=Success
Fd 2046: -1, err=Success
Fd 2047: -1, err=Success

That doesn't look right. On my system I get

Fd 3: -1, err=Bad file descriptor

It looks like fcntl is not setting errno correctly when it fails. I suggest you look into that. It could be a bug in the libc you're using.

Anyway I think this is pretty clearly not an rr bug.

Thank you.

Navigation Menu

Repository navigation

rr record & reply fails on windriver linux #2436

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

rr record & reply fails on windriver linux #2436

Description

Activity

rocallahan commented on Feb 2, 2020

syrajendra commented on Feb 2, 2020

rocallahan commented on Feb 2, 2020

syrajendra commented on Feb 3, 2020

rocallahan commented on Feb 3, 2020

rocallahan commented on Feb 3, 2020

syrajendra commented on Feb 3, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions