RP 0.45.RC1 failure on comet

[vivek-bala]

Script: 00_getting_started.py
Resource: comet
Full log: https://gist.github.com/vivek-bala/0d0d4ee7662c2e8d2ed9ebb744cc295e

Interessting part of the log:

2017-02-13 17:41:59,081: radical.saga.pty    : MainProcess                     : MainThread     : DEBUG   : Traceback (most recent call last):
  File "/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_process.py", line 787, in find
    data += self.read (timeout=_POLLDELAY)
  File "/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_process.py", line 679, in read
    % (e, self.tail))
NoSuccess: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
) (/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_process.py +679 (read)  :  % (e, self.tail)))

2017-02-13 17:41:59,135: radical.saga.pty    : MainProcess                     : MainThread     : ERROR   : read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
) ((ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
)) (/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_exceptions.py +40 (translate_exception)  :  e = se.BadParameter (cmsg))
Traceback (most recent call last):
  File "/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 263, in _initialize_pty
    n, match = pty_shell.find (prompt_patterns, delay)
  File "/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_process.py", line 790, in find
    raise ptye.translate_exception (e, "(%s)" % data)
BadParameter: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
) ((ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
)) (/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_exceptions.py +40 (translate_exception)  :  e = se.BadParameter (cmsg))
caught Exception: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
) ((ssh: Could not resolve hostname comet.sdsc.xsede.org: Name or service not known
)) (/home/vivek/Research/ves/test_rp/local/lib/python2.7/site-packages/saga/utils/pty_exceptions.py +40 (translate_exception)  :  e = se.BadParameter (cmsg))

This error is reproducible. The sandbox on comet just contains only the bootstrap_1.sh script.

[andre-merzky]

That is a strange one though: can you try to run the following on command line, please:

$ host comet.sdsc.xsede.org
comet.sdsc.xsede.org is an alias for comet.sdsc.edu.
comet.sdsc.edu has address 198.202.113.252
comet.sdsc.edu has address 198.202.113.253

If you see the same, please run the test again. If you see something different, please post the output.

Thanks!

[vivek-bala]

$ host comet.sdsc.xsede.org
comet.sdsc.xsede.org is an alias for comet.sdsc.edu.
comet.sdsc.edu has address 198.202.113.253
comet.sdsc.edu has address 198.202.113.252
comet.sdsc.edu mail is handled by 10 postal.sdsc.edu.

[vivek-bala]

I get the same error.

[andre-merzky]

From what machine are you running, Vivek? Can you try to run from the radical server, and compare results? I am not saying this should not work out of the box - but this looks more like a network issue than anything else to me?

[vivek-bala]

Hmmm. Sure I'll try it out from our VM.

[vivek-bala]

From the radical VM, I don't see the above error.

But, the CUs start executing and then fail with the following error:

/home/marksant/openmpi/installed/rhc/bin/orterun: Error: unknown option "--hnp"
Type '/home/marksant/openmpi/installed/rhc/bin/orterun --help' for usage.

[andre-merzky]

Hmm, that is an ORTE deployment error it seems, and indeed different from what you had before. Can you please open a new ticket for this one? I'll need to look into it. Thanks!

Not sure how to proceed with your original problem: your logs indicate that the ssh login to comet works well, but that the file transfer stalls and times out. Let me cobble up a couple of commands to run some tests...

[vivek-bala]

I opened a new ticket for the orte failure (#1218). Let me know how I can debug the original error.

[andre-merzky]

For the original error: can you please set RADICAL_SAGA_PTY_VERBOSE=DEBUG, create a new logfile, and attach it? Thanks!

[vivek-bala]

Ok, I can't reproduce this error anymore. I'll close this for now. Will reopen if I run into this issue again.