Document common cluster specific (but not job scheduler specific) quirks with work-arounds if available

[lesteve]

I wish there were some list of cluster configuration quirks (that are not job scheduler specific) and possible work-arounds (when there are some) somewhere in the doc (I was not aware of limitation of TCP/IP connection limitations between login and compute nodes in some clusters until a few days ago). Here are a rough list off the top of my head:

• submit_command not available on the compute nodes, e.g. For loop never ending #333. Possible work-around: For loop never ending #333 (comment) (I never tried it myself). This is the case for all the OAR clusters I know about, i.e. the submit command is never available on the compute nodes so in principle I could test this idea.
• TCP/IP restrictions between login and compute nodes e.g. Dask JobQueue and TCP connections between login and compute nodes #354 and Specify a cluster scheduler listener port? #355. Possible work-around: start the main script / notebook in an interactive node with all the additional pain and limitations this entails, see Dask JobQueue and TCP connections between login and compute nodes #354 (comment) for the one I know about.
• non uniform network interfaces on login and compute nodes. I guess same work-around as TCP/IP restriction would work but not a great work-around.

Please add more if you know more off the top of your head.

cc @mrocklin @guillaumeeb @jhamman

[mrocklin]

There are a few here: https://blog.dask.org/2019/08/28/dask-on-summit

…

On Wed, Oct 16, 2019 at 10:22 PM Loïc Estève ***@***.***> wrote: I wish there were some list of cluster configuration quirks (that are not job scheduler specific) and possible work-arounds (when there are some) somewhere in the doc (I was not aware of limitation of TCP/IP connection limitations between login and compute nodes in some clusters until a few days ago). Here are a rough list off the top of my head: - submit_command not available on the compute nodes, e.g. #333 <#333>. Possible work-around: #333 (comment) <#333 (comment)> (I never tried it myself). This is the case for all the OAR clusters I know about, i.e. the submit command is never available on the compute nodes - TCP/IP restrictions between login and compute nodes e.g. #354 <#354> and #355 <#355>. Possible work-around: start the main script / notebook in an interactive node with all the additional pain and limitations this entails, see #354 (comment) <#354 (comment)> for the one I know about. - non uniform network interfaces on login and compute nodes. I guess same work-around as TCP/IP restriction would work but not a great work-around. Please add more if you know more off the top of your head. cc @mrocklin <https://github.com/mrocklin> @guillaumeeb <https://github.com/guillaumeeb> @jhamman <https://github.com/jhamman> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#356?email_source=notifications&email_token=AACKZTHMD46MLHRBJ4XMU2LQO7LAXA5CNFSM4JBTULS2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HSKVIJQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTHPA26ZL37UUGCRI3DQO7LAXANCNFSM4JBTULSQ> .

[lesteve]

I reread your blog post to be sure, but I don't think the ones in your blogpost are the one about bsub < job_script vs bsub job_script unless I missed something (aka use_stdin=True vs `use_stdin=False)

Sorry wrong issue, ignore this.

[lesteve]

There are a few here: blog.dask.org/2019/08/28/dask-on-summit

Good point!