Merge branch 'main' of https://github.com/aniskhan25/hmsc-hpc into main

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ on:
 jobs:
   test-updaters:
 
-    runs-on: macos-latest
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.11"]
@@ -23,7 +23,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
-          cache-dependency-path: 'requirements_dev.txt'
+          cache-dependency-path: 'requirements.txt'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

examples/basic_example/example.Rmd

@@ -5,7 +5,7 @@ output:
     df_print: paged
 ---
 
-This notebook demonstrates the concept of using Hmsc-HPC extension for `Hmsc` package. Unlike the core `Hmsc`, the Hmsc-HPC extension is written in Python programming language and is executed with Python interpreter. Hence, before a user can use it, a proper Python installation is required.
+This notebook demonstrates the concept of using Hmsc-HPC extension for `Hmsc` package. Unlike the core `Hmsc`, the Hmsc-HPC extension is written in Python programming language and is executed with Python interpreter. Hence, before a user can use it, a proper Python installation is required. However, detailed knowledge of Python is not needed to use Hmsc-HPC, it's use is transparent once Python is installed.
 
 # Preparing Python environment
 
@@ -17,22 +17,19 @@ This notebook demonstrates the concept of using Hmsc-HPC extension for `Hmsc` pa
 
 ### 1. Finding Python installation
 
-Depending on your hardware, operating system and user rights, the set of steps to acquire and configure a proper Python distribution may greatly vary. Thus, we would like to relay the installation process itself either to one of multitude guides available on the web, or to the IT support that manages your device. This section merely checks that a Python installation can be found.
-
-Please test the next chunk of code that tries to check version of Python available in you system:
+Please run the next chunk of code to check the version of Python available in you system:
 
 ```{r}
 system_python = "python3"
 # system_python = "/Users/username/opt/anaconda3/envs/tf/bin/python3"
 system2(system_python, "--version")
 ```
 
-If the Python distribution in your system is configured well, then the code shall print the version of Python. If this is failing, then you are likely missing Python altogether, or its path is not configured correctly. In some cases you may have several distributions of Python available, and then you shall either explicitly specify the path to the desired Python distribution --- as exemplified in the commented-out `system_python = ...` line above.
-
-### 2. Creating a new virtual environment
+If the Python distribution in your system is configured well, then the code shall print the version of Python. If this is failing, then you are likely missing Python altogether, or its path is not configured correctly. If this is the case, please follow the instructions at https://wiki.python.org/moin/BeginnersGuide/Download for installing Python for the operating system (Linux, Mac, Windows) of your computer.
 
-Please note that this notebook is configured **NOT** to execute the codes that install any software or packages --- both in this and next sections during whole notebook execution. Please execute them manually one by one if needed.
+In some cases you may have several distributions of Python available, and then you shall either explicitly specify the path to the desired Python distribution --- as exemplified in the commented-out `system_python = ...` line above.
 
+### 2. Creating a new virtual environment
 
 The next line creates an empty virtual environment where we will set up Hmsc-HPC:
 
@@ -83,7 +80,10 @@ Your Python setup is working correctly if the code does not produce any errors.
 
 ### Troubleshooting
 
-If the above check and the detailed instructions do not work for you, please consult one of multitude guides available on the web. You can configure Python within R in multiple ways - for instance, `reticulate` package features several functions aimed to achieve that or you can prepare and activate the correct Python environment outside R.
+If the above check do not work for you, please follow [Detailed instructions] including installing a clean Python environment to start with. Alternatively, you can prepare and activate the Python environment outside R and execute the same Python commands there.
+
+Hmsc-HPC extension relies on Tensorflow for GPU acceleration, and the code will utilize CPUs if a GPU-enabled version of Tensorflow is not available. Enabling GPU acceleration in Tensorflow depends on your operating system as explained in https://www.tensorflow.org/install/pip.
+
 
 # Setting up a toy Hmsc model
 

examples/basic_example/example.nb.html

@@ -1756,7 +1756,9 @@ <h1 class="title toc-ignore">Basic Hmsc-HPC usage example</h1>
 for <code>Hmsc</code> package. Unlike the core <code>Hmsc</code>, the
 Hmsc-HPC extension is written in Python programming language and is
 executed with Python interpreter. Hence, before a user can use it, a
-proper Python installation is required.</p>
+proper Python installation is required. However, detailed knowledge of
+Python is not needed to use Hmsc-HPC, it's use is transparent once
+Python is installed.</p>
 <div id="preparing-python-environment" class="section level1">
 <h1>Preparing Python environment</h1>
 <p><strong>If you are familiar with using Python within R</strong>, then
@@ -1771,14 +1773,8 @@ <h1>Preparing Python environment</h1>
 <h2>Detailed instructions</h2>
 <div id="finding-python-installation" class="section level3">
 <h3>1. Finding Python installation</h3>
-<p>Depending on your hardware, operating system and user rights, the set
-of steps to acquire and configure a proper Python distribution may
-greatly vary. Thus, we would like to relay the installation process
-itself either to one of multitude guides available on the web, or to the
-IT support that manages your device. This section merely checks that a
-Python installation can be found.</p>
-<p>Please test the next chunk of code that tries to check version of
-Python available in you system:</p>
+<p>Please run the next chunk of code to check the version of Python
+available in you system:</p>
 <!-- rnb-text-end -->
 <!-- rnb-chunk-begin -->
 <!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuc3lzdGVtX3B5dGhvbiA9IFwicHl0aG9uM1wiXG4jIHN5c3RlbV9weXRob24gPSBcIi9Vc2Vycy91c2VybmFtZS9vcHQvYW5hY29uZGEzL2VudnMvdGYvYmluL3B5dGhvbjNcIlxuc3lzdGVtMihzeXN0ZW1fcHl0aG9uLCBcIi0tdmVyc2lvblwiKVxuYGBgIn0= -->
@@ -1794,7 +1790,11 @@ <h3>1. Finding Python installation</h3>
 <p>If the Python distribution in your system is configured well, then
 the code shall print the version of Python. If this is failing, then you
 are likely missing Python altogether, or its path is not configured
-correctly. In some cases you may have several distributions of Python
+correctly. If this is the case, please follow the instructions at
+<a href="https://wiki.python.org/moin/BeginnersGuide/Download">https://wiki.python.org/moin/BeginnersGuide/Download</a>
+for installing Python for the operating system (Linux, Mac, Windows) of
+your computer.</p>
+<p>In some cases you may have several distributions of Python
 available, and then you shall either explicitly specify the path to the
 desired Python distribution — as exemplified in the commented-out
 <code>system_python = ...</code> line above.</p>
@@ -2066,12 +2066,16 @@ <h2>Checking Python environment</h2>
 any errors.</p>
 <div id="troubleshooting" class="section level3">
 <h3>Troubleshooting</h3>
-<p>If the above check and the detailed instructions do not work for you,
-please consult one of multitude guides available on the web. You can
-configure Python within R in multiple ways - for instance,
-<code>reticulate</code> package features several functions aimed to
-achieve that or you can prepare and activate the correct Python
-environment outside R.</p>
+<p>If the above check do not work for you, please follow
+<a href="#detailed-instructions">Detailed instructions</a>
+including installing a clean Python environment to start with.
+Alternatively, you can prepare and activate the Python environment
+outside R and execute the same Python commands there.</p>
+<p>Hmsc-HPC extension relies on Tensorflow for GPU acceleration, and
+the code will utilize CPUs if a GPU-enabled version of Tensorflow is
+not available. Enabling GPU acceleration in Tensorflow depends on your
+operating system as explained in
+<a href="https://www.tensorflow.org/install/pip">https://www.tensorflow.org/install/pip</a>.</p>
 </div>
 </div>
 </div>

hmsc/test/updateLambdaPriors_test.py

@@ -135,7 +135,7 @@ def test_updateLambdaPriors():
     PsiList, DeltaList = updateLambdaPriors(params, rLHyperparams)
 
     for r in range(modelDims["nr"]):
-        assert_allclose(tf.reduce_mean(PsiList[r]), tf.reduce_mean(PsiListTrue[r]), atol=1.3)
+        assert_allclose(tf.reduce_mean(PsiList[r]), tf.reduce_mean(PsiListTrue[r]), atol=1.5)
         assert_allclose(tf.reduce_mean(DeltaList[r]), tf.reduce_mean(DeltaListTrue[r]), atol=1.0)
 
 def test_updateLambdaPriors_shape():

hmsc/test/updatewRRRPriors_test.py

@@ -160,8 +160,8 @@ def test_updatewRRRPriors():
 
     PsiRRR, DeltaRRR = updatewRRRPriors(params, modelDims, priorHyperparams)
 
-    assert_allclose(tf.reduce_mean(PsiRRR), tf.reduce_mean(PsiRRRTrue), atol=1.0)
-    assert_allclose(tf.reduce_mean(DeltaRRR), tf.reduce_mean(DeltaRRRTrue), atol=2.0)
+    assert_allclose(tf.reduce_mean(PsiRRR), tf.reduce_mean(PsiRRRTrue), atol=1.5)
+    assert_allclose(tf.reduce_mean(DeltaRRR), tf.reduce_mean(DeltaRRRTrue), atol=3.0)
 
 
 def test_updatewRRRPriors_shape():

requirements_dev.txt

@@ -2,7 +2,7 @@ numpy==1.26.2
 pandas==2.1.3
 pyreadr==0.5.0
 scipy==1.11.4
-tensorflow==2.15.0
-tensorflow-probability==0.23.0
+tensorflow==2.16.1
+tensorflow-probability==0.24.0
 ujson==5.8.0
 

setup.py

@@ -10,8 +10,8 @@
         'pandas',
         'pyreadr',
         'scipy',
-        'tensorflow==2.15.0',
-        'tensorflow-probability==0.23.0',
+        'tensorflow[and-cuda]',
+        'tensorflow-probability[tf]',
         'ujson',
     ]
 )