Parsl deployment on dev-qgnet fails

[trey-stafford]

Currently, running on dev-qgnet fails.

• run.py hangs
• The pod created in the dev-qgnet cluster goes into a restart loop with this log/error message output:

  Downloading parsl-2024.4.8-py3-none-any.whl.metadata (4.0 kB)
Collecting pyzmq>=17.1.2 (from parsl)
  Downloading pyzmq-25.1.2-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (4.9 kB)
Collecting typeguard!=3.*,<5,>=2.10 (from parsl)
  Downloading typeguard-4.2.1-py3-none-any.whl.metadata (3.7 kB)
Collecting typing-extensions<5,>=4.6 (from parsl)
  Downloading typing_extensions-4.11.0-py3-none-any.whl.metadata (3.0 kB)
Collecting globus-sdk (from parsl)
  Downloading globus_sdk-3.39.0-py3-none-any.whl.metadata (3.2 kB)
Collecting dill (from parsl)
  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Collecting tblib (from parsl)
  Downloading tblib-3.0.0-py3-none-any.whl.metadata (25 kB)
Collecting requests (from parsl)
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting paramiko (from parsl)
  Downloading paramiko-3.4.0-py3-none-any.whl.metadata (4.4 kB)
Collecting psutil>=5.5.1 (from parsl)
  Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting setproctitle (from parsl)
  Downloading setproctitle-1.3.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.9 kB)
Collecting pyjwt<3.0.0,>=2.0.0 (from pyjwt[crypto]<3.0.0,>=2.0.0->globus-sdk->parsl)
  Downloading PyJWT-2.8.0-py3-none-any.whl.metadata (4.2 kB)
Collecting cryptography!=3.4.0,>=3.3.1 (from globus-sdk->parsl)
  Downloading cryptography-42.0.5-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.3 kB)
Collecting charset-normalizer<4,>=2 (from requests->parsl)
  Downloading charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->parsl)
  Downloading idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests->parsl)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests->parsl)
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting bcrypt>=3.2 (from paramiko->parsl)
  Downloading bcrypt-4.1.2-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (9.5 kB)
Collecting pynacl>=1.5 (from paramiko->parsl)
  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (8.6 kB)
Collecting cffi>=1.12 (from cryptography!=3.4.0,>=3.3.1->globus-sdk->parsl)
  Downloading cffi-1.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.12->cryptography!=3.4.0,>=3.3.1->globus-sdk->parsl)
  Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading parsl-2024.4.8-py3-none-any.whl (522 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 522.5/522.5 kB 2.8 MB/s eta 0:00:00
Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 288.2/288.2 kB 4.5 MB/s eta 0:00:00
Downloading pyzmq-25.1.2-cp312-cp312-manylinux_2_28_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 8.7 MB/s eta 0:00:00
Downloading typeguard-4.2.1-py3-none-any.whl (34 kB)
Downloading typing_extensions-4.11.0-py3-none-any.whl (34 kB)
Downloading dill-0.3.8-py3-none-any.whl (116 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 1.2 MB/s eta 0:00:00
Downloading globus_sdk-3.39.0-py3-none-any.whl (301 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.8/301.8 kB 5.6 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 2.0 MB/s eta 0:00:00
Downloading paramiko-3.4.0-py3-none-any.whl (225 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 225.9/225.9 kB 943.6 kB/s eta 0:00:00
Downloading setproctitle-1.3.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB)
Downloading tblib-3.0.0-py3-none-any.whl (12 kB)
Downloading bcrypt-4.1.2-cp39-abi3-manylinux_2_28_x86_64.whl (698 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 698.9/698.9 kB 6.3 MB/s eta 0:00:00
Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 3.6 MB/s eta 0:00:00
Downloading charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 141.9/141.9 kB 4.5 MB/s eta 0:00:00
Downloading cryptography-42.0.5-cp39-abi3-manylinux_2_28_x86_64.whl (4.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 11.5 MB/s eta 0:00:00
Downloading idna-3.7-py3-none-any.whl (66 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 kB 631.8 kB/s eta 0:00:00
Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB)
Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 856.7/856.7 kB 6.7 MB/s eta 0:00:00
Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 1.8 MB/s eta 0:00:00
Downloading cffi-1.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (477 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 477.6/477.6 kB 11.2 MB/s eta 0:00:00
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 850.4 kB/s eta 0:00:00
Installing collected packages: urllib3, typing-extensions, tblib, setproctitle, pyzmq, pyjwt, pycparser, psutil, idna, dill, charset-normalizer, certifi, bcrypt, typeguard, requests, cffi, pynacl, cryptography, paramiko, globus-sdk, parsl
Successfully installed bcrypt-4.1.2 certifi-2024.2.2 cffi-1.16.0 charset-normalizer-3.3.2 cryptography-42.0.5 dill-0.3.8 globus-sdk-3.39.0 idna-3.7 paramiko-3.4.0 parsl-2024.4.8 psutil-5.9.8 pycparser-2.22 pyjwt-2.8.0 pynacl-1.5.0 pyzmq-25.1.2 requests-2.31.0 setproctitle-1.3.3 tblib-3.0.0 typeguard-4.2.1 typing-extensions-4.11.0 urllib3-2.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Failed to find a viable address to connect to interchange. Exiting
/bin/bash: -c: line 4: syntax error near unexpected token `;'
/bin/bash: -c: line 4: `;'```

[trey-stafford]

The most important bit of the failure given in the message above appears to be:

Failed to find a viable address to connect to interchange. Exiting

Which stems from this parsl code

Pods stuck in a restart loop should be manually cleaned-up with kubectl delete pod <pod_name>.

[trey-stafford]

We managed to work around the interchange issue by submitting our parsl code to the cluster as a k8s Job.

WIP branch: https://github.com/QGreenland-Net/parsl-exploration/tree/use-custom-image