You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).
I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934
I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss)
I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001
I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332
I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
PC: @ 0x7f1f2c367f56 (unknown)
PC: @ 0x7fc173393f56 (unknown)
PC: @ 0x7fef15a8df56 (unknown)
PC: @ 0x7ff6e251ef56 (unknown)
mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).
The text was updated successfully, but these errors were encountered:
I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).
I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934
I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss)
I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001
I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332
I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
*** Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date ***
PC: @ 0x7f1f2c367f56 (unknown)
PC: @ 0x7fc173393f56 (unknown)
PC: @ 0x7fef15a8df56 (unknown)
PC: @ 0x7ff6e251ef56 (unknown)
mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).
The text was updated successfully, but these errors were encountered: