Merge branch 'devel_nodb_2' into fix/reqs_config

radical-cybertools · Sep 26, 2023 · cd92bbb · cd92bbb
2 parents 953c510 + ef06deb
commit cd92bbb
Show file tree

Hide file tree

Showing 4 changed files with 43 additions and 48 deletions.
diff --git a/.github/workflows/run-rp-notebook.yml b/.github/workflows/run-rp-notebook.yml
@@ -41,6 +41,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
       - name: Install dependencies
         run: |
+          sudo apt update -y && sudo apt install -y mpich
           python -m venv testenv
           . testenv/bin/activate
           python -m pip install --upgrade pip setuptools wheel

diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -3,8 +3,14 @@ version: 2
 
 formats: [htmlzip]
 
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.7"
+  apt_packages:
+    - mpich
+
 python:
-  version: 3.7
   install:
     - requirements: requirements-docs.txt
     - method: pip

diff --git a/docs/source/tutorials/describing_tasks.ipynb b/docs/source/tutorials/describing_tasks.ipynb
@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "67521807",
    "metadata": {},
@@ -22,58 +21,51 @@
     "\n",
     "</div>\n",
     "\n",
-    "First, some preparatory work for the tutorial. We import some modules and set some variables. Note that we `import radical.pilot as rp` so to abbreviate future API calls. "
+    "Let's have a quick check that we have MPI launch method installed. "
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c8b8387d",
+   "id": "567c0f26-3e35-44d3-a81a-ab89a79a3dcd",
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
-    "import sys\n",
-    "import pprint\n",
-    "\n",
-    "# do not use animated output in notebooks\n",
-    "os.environ['RADICAL_REPORT_ANIME'] = 'False'\n",
-    "\n",
-    "import radical.pilot as rp\n",
     "import radical.utils as ru\n",
     "\n",
-    "# determine the path of the currently active virtualenv to simplify some examples below\n",
-    "ve_path = os.path.dirname(os.path.dirname(ru.which('python3')))\n",
-    "display(ve_path)\n"
+    "mpi_lm_exists = bool(ru.which(['mpirun', 'mpiexec']))"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c2c6a8bf-3128-42a6-932e-32fdf9f87c55",
+   "id": "ec11a436-b9a8-4706-a126-d5f1ba19bd41",
    "metadata": {},
    "source": [
-    "<div class=\"alert alert-warning\">\n",
-    "    \n",
-    "__Warning:__ MPI ranks can only be used when the `mpiexec` or `mpirun` commands are available - we test this here to avoid runtime errors\n",
-    "\n",
-    "</div>"
+    "First, some preparatory work for the tutorial. We import some modules and set some variables. Note that we `import radical.pilot as rp` so to abbreviate future API calls. "
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a6983867-7111-4cbb-ac66-def4075765e4",
+   "id": "c8b8387d",
    "metadata": {},
    "outputs": [],
    "source": [
-    "max_ranks = 4\n",
+    "import os\n",
+    "import sys\n",
+    "import pprint\n",
+    "\n",
+    "# do not use animated output in notebooks\n",
+    "os.environ['RADICAL_REPORT_ANIME'] = 'False'\n",
+    "\n",
+    "import radical.pilot as rp\n",
     "\n",
-    "if not ru.which(['mpirun', 'mpiexec']):\n",
-    "    max_ranks = 1"
+    "# determine the path of the currently active virtualenv to simplify some examples below\n",
+    "ve_path = os.path.dirname(os.path.dirname(ru.which('python3')))\n",
+    "display(ve_path)"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "97ab1560",
    "metadata": {},
@@ -113,7 +105,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "1ce411cb",
    "metadata": {},
@@ -144,11 +135,10 @@
    "source": [
     "# create a minimal executable task\n",
     "td   = rp.TaskDescription({'executable': '/bin/date'})\n",
-    "task = tmgr.submit_tasks(td)\n"
+    "task = tmgr.submit_tasks(td)"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "03112275",
    "metadata": {},
@@ -167,7 +157,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "9efbbe7a",
    "metadata": {},
@@ -210,11 +199,10 @@
     "report.plain('pilot_sandbox   : %s\\n' % task.pilot_sandbox)\n",
     "report.plain('task_sandbox    : %s\\n' % task.task_sandbox)\n",
     "report.plain('client_sandbox  : %s\\n' % task.client_sandbox)\n",
-    "report.plain('metadata        : %s\\n' % task.metadata)\n"
+    "report.plain('metadata        : %s\\n' % task.metadata)"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "7273c6ea",
    "metadata": {},
@@ -254,7 +242,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "187dbca6",
    "metadata": {},
@@ -277,7 +264,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "0cc12709",
    "metadata": {},
@@ -296,20 +282,19 @@
     "\n",
     "for task in tasks:\n",
     "    if task.state == rp.FAILED:\n",
-    "        print('%s: %s: %s' % (task.uid, task.description['arguments'], task.stderr.strip()))\n"
+    "        print('%s: %s: %s' % (task.uid, task.description['arguments'], task.stderr.strip()))"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "82299910",
    "metadata": {},
    "source": [
     "## MPI Tasks and Task Resources\n",
     "\n",
-    "So far, we run single-core tasks.  The most common way for application to utilize multiple cores and nodes on HPC machines is to use MPI as communication layer which coordinates multiple application processes, i.e., MPI ranks.  In fact, the notion of `ranks` is central to RP's `TaskDescription` class. All MPI ranks will be near-exact copies of each other: they run in the same work directory and the same `environment`, are defined by the same `executable` and `arguments`, get the same amount of resources allocated, etc. Notable exceptions are:\n",
+    "So far, we run single-core tasks.  The most common way for application to utilize multiple cores and nodes on HPC machines is to use MPI as a communication layer, which coordinates multiple application processes, i.e., MPI ranks.  In fact, the notion of `ranks` is central to RP's `TaskDescription` class. All MPI ranks will be near-exact copies of each other: they run in the same work directory and the same `environment`, are defined by the same `executable` and `arguments`, get the same amount of resources allocated, etc. Notable exceptions are:\n",
     "\n",
-    "  - Rank processes may run on different nodes;\n",
+    "  - rank processes may run on different nodes;\n",
     "  - rank processes can communicate via MPI;\n",
     "  - each rank process obtains a unique rank ID.\n",
     "\n",
@@ -337,6 +322,12 @@
     "\n",
     "__Note:__ No core pinning is performed on localhost. Thus, tasks see all CPU cores as available to them.  However, the `THREADS` information still reports the correct number of assigned CPU cores.\n",
     "\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-info\">\n",
+    "\n",
+    "__Note:__ If there is no MPI launch method installed, then we will proceed with a single rank.\n",
+    "\n",
     "</div>"
    ]
   },
@@ -349,9 +340,10 @@
    "source": [
     "tds = list()\n",
     "for n in range(4):\n",
+    "    ranks = (n + 1) if mpi_lm_exists else 1\n",
     "    tds.append(rp.TaskDescription({'executable'    : ve_path + '/bin/radical-pilot-hello.sh',\n",
     "                                   'arguments'     : [n + 1], \n",
-    "                                   'ranks'         : min(max_ranks, (n + 1)), \n",
+    "                                   'ranks'         : ranks, \n",
     "                                   'cores_per_rank': (n + 1),\n",
     "                                   'threading_type': rp.OpenMP}))\n",
     "    report.progress()\n",
@@ -362,11 +354,10 @@
     "tmgr.wait_tasks([task.uid for task in tasks])\n",
     "\n",
     "for task in tasks:\n",
-    "    print('--- %s:\\n%s\\n' % (task.uid, task.stdout.strip()))\n"
+    "    print('--- %s:\\n%s\\n' % (task.uid, task.stdout.strip()))"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "420ed233",
    "metadata": {},
@@ -409,7 +400,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "a4bb97c2",
    "metadata": {},
@@ -424,7 +414,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "200d8813",
    "metadata": {},
@@ -467,7 +456,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "70d849d8",
    "metadata": {},
@@ -504,11 +492,10 @@
     "                        })\n",
     "task = tmgr.submit_tasks(td)\n",
     "tmgr.wait_tasks([task.uid])\n",
-    "print(task.stdout)\n"
+    "print(task.stdout)"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "78f3f8a7",
    "metadata": {},
@@ -571,7 +558,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.5"
+   "version": "3.9.13"
   },
   "varInspector": {
    "cols": {

diff --git a/src/radical/pilot/raptor/worker.py b/src/radical/pilot/raptor/worker.py
@@ -48,7 +48,8 @@ def __init__(self, manager, rank, raptor_id):
                                debug=self._cfg.debug_lvl,
                                targets=self._cfg.log_tgt,
                                path=self._cfg.path)
-        self._prof = ru.Profiler(name=self._uid, ns='radical.pilot.worker',
+        self._prof = ru.Profiler(name='%s.%04d' % (self._uid, self._rank),
+                                 ns='radical.pilot.worker',
                                  path=self._cfg.path)
 
         # register for lifetime management messages on the control pubsub