Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exited on signal 7 (Bus error). #265

Open
Hiker01 opened this issue Apr 4, 2019 · 0 comments
Open

exited on signal 7 (Bus error). #265

Hiker01 opened this issue Apr 4, 2019 · 0 comments

Comments

@Hiker01
Copy link

Hiker01 commented Apr 4, 2019

I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).

I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934
I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss)
I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001
I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332
I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
PC: @ 0x7f1f2c367f56 (unknown)
PC: @ 0x7fc173393f56 (unknown)
PC: @ 0x7fef15a8df56 (unknown)
PC: @ 0x7ff6e251ef56 (unknown)

mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant