Merge pull request #102 from BlackHolePerturbationToolkit/download

Change data download server from Zenodo to download.bhptoolkit.org

README.md

@@ -26,7 +26,7 @@ To install this software for CPU usage, you need [gsl >2.0](https://www.gnu.org/
 
 To install this software for use with NVIDIA GPUs (compute capability >2.0), you need the [CUDA toolkit](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) and [CuPy](https://cupy.chainer.org/). The CUDA toolkit must have cuda version >8.0. Be sure to properly install CuPy within the correct CUDA toolkit version. Make sure the nvcc binary is on `$PATH` or set it as the `CUDA_HOME` environment variable.
 
-There are a set of files required for total use of this package. They will download automatically the first time they are needed. Files are generally under 10MB. However, there is a 100MB file needed for the slow waveform and the bicubic amplitude interpolation. This larger file will only download if you run either of those two modules. The files are hosted on [Zenodo](https://zenodo.org/record/3981654#.XzS_KRNKjlw).
+There are a set of files required for total use of this package. They will download automatically the first time they are needed. Files are generally under 10MB. However, there is a 100MB file needed for the slow waveform and the bicubic amplitude interpolation. This larger file will only download if you run either of those two modules. The files are hosted on the [Black Hole Perturbation Toolkit Download Server](https://download.bhptoolkit.org/few/data).
 
 ### Installing
 
@@ -80,13 +80,13 @@ Please contact the developers if the installation does not work.
 1) Create a virtual environment.
 
 ```
-conda create -n few_env -c conda-forge gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests matplotlib python=3.7
+conda create -n few_env -c conda-forge gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests rich matplotlib python=3.7
 conda activate few_env
 ```
 
     If on MACOSX, substitute `gcc_linux-64` and `gxx_linus-64` with `clang_osx-64` and `clangxx_osx-64`.
 
-    If you want a faster install, you can install the python packages (numpy, Cython, scipy, tqdm, jupyter, ipython, h5py, requests, matplotlib) with pip.
+    If you want a faster install, you can install the python packages (numpy, Cython, scipy, tqdm, jupyter, ipython, h5py, requests, rich, matplotlib) with pip.
 
 2) Clone the repository.
 

docs/source/README.rst

@@ -76,7 +76,7 @@ will download automatically the first time they are needed. Files are
 generally under 10MB. However, there is a 100MB file needed for the slow
 waveform and the bicubic amplitude interpolation. This larger file will
 only download if you run either of those two modules. The files are
-hosted on `Zenodo <https://zenodo.org/record/3981654#.XzS_KRNKjlw>`__.
+hosted on the `Black Hole Perturbation Toolkit Download Server <https://download.bhptoolkit.org/few/data>`__.
 
 Installing
 ~~~~~~~~~~
@@ -146,14 +146,14 @@ More Customized Installation (legacy)
 
 ::
 
-   conda create -n few_env -c conda-forge gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests matplotlib python=3.7
+   conda create -n few_env -c conda-forge gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests rich matplotlib python=3.7
    conda activate few_env
 
 ::
 
    If on MACOSX, substitute `gcc_linux-64` and `gxx_linus-64` with `clang_osx-64` and `clangxx_osx-64`.
 
-   If you want a faster install, you can install the python packages (numpy, Cython, scipy, tqdm, jupyter, ipython, h5py, requests, matplotlib) with pip.
+   If you want a faster install, you can install the python packages (numpy, Cython, scipy, tqdm, jupyter, ipython, h5py, requests, rich, matplotlib) with pip.
 
 2) Clone the repository.
 

few/amplitude/interp2dcubicspline.py

@@ -58,7 +58,7 @@ def __init__(self, **kwargs):
         AmplitudeBase.__init__(self, **kwargs)
 
         # check if you have the necessary file
-        # it will download from Zenodo if the user does not have it.
+        # it will download from download.bhptoolkit.org if the user does not have it.
         few_dir = dir_path + "/../../"
 
         fp = "Teuk_amps_a0.0_lmax_10_nmax_30_new.h5"

few/trajectory/inspiral.py

@@ -139,7 +139,7 @@ def __init__(
                 check_for_file_download(fp, few_dir)
             except FileNotFoundError:
                 raise ValueError(
-                    f"File required for this ODE ({fp}) was not found in the proper folder ({few_dir + 'few/files/'}) or on zenodo."
+                    f"File required for this ODE ({fp}) was not found in the proper folder ({few_dir + 'few/files/'}) or on download.bhptoolkit.org."
                 )
 
         self.test_new_version = test_new_version

few/utils/utility.py

@@ -20,6 +20,7 @@
 import os
 import subprocess
 import warnings
+from rich.progress import track
 
 import numpy as np
 from scipy.interpolate import CubicSpline
@@ -749,10 +750,10 @@ def get_mu_at_t(
 
 
 def check_for_file_download(fp, few_dir, version_string=None):
-    """Download files direct from zenodo.
+    """Download files direct from download.bhptoolkit.org.
 
-    This function downloads the files from zenodo as they are needed. They are
-    downloaded based on the associated record for each version (`record_by_version`).
+    This function downloads the files from download.bhptoolkit.org as they are needed. They are
+    downloaded based on the associated Zenodo record for each version (`record_by_version`).
 
     The version is determined from the `__version__` attribute of `few` unless
     a version string is provided.
@@ -788,29 +789,31 @@ def check_for_file_download(fp, few_dir, version_string=None):
         os.mkdir(few_dir + "few/files/")
 
     # check if the file is in the files filder
-    # if not, download it from zenodo
+    # if not, download it from download.bhptoolkit.org
     if fp not in os.listdir(few_dir + "few/files/"):
-        warnings.warn(
-            "The file {} did not open sucessfully. It will now be downloaded to the proper location.".format(
-                fp
-            )
-        )
+        print("Data file " + fp + " not found. Downloading now.")
 
         # get record number based on version
         # record = record_by_version.get(version_string)
 
         # temporary fix
         record = 3981654
 
-        # url to zenodo API
-        url = "https://zenodo.org/record/" + str(record) + "/files/" + fp
-
-        # run wget from terminal to get the folder
-        # download to proper location
-        subprocess.run(["wget", "--no-check-certificate", url])
-
-        # move it into the files folder
-        os.rename(fp, few_dir + "few/files/" + fp)
+        # url to download from with Zenodo fallback in case of failure
+        url = "https://download.bhptoolkit.org/few/data/" + str(record) + "/" + fp
+        zenodourl = "https://zenodo.org/record/" + str(record) + "/files/" + fp
+
+        # download the file
+        response = requests.get(url, stream=True)
+        if response.ok != True:
+          response = requests.get(zenodourl, stream=True)
+
+        # Save the file to the files folder, downloading 8KB at a time
+        with open(few_dir + "few/files/" + fp, mode="wb") as file:
+          filesize = int(response.headers.get('content-length'))
+          csize = 2**15
+          for chunk in track(response.iter_content(chunk_size = csize), description="Downloading "+fp, total=filesize/csize):
+            file.write(chunk)
 
 
 def wrapper(*args, **kwargs):

install.sh

@@ -92,14 +92,14 @@ if [[ "$OSTYPE" == "darwin"* ]]; then
     machine=$(uname -m)
     if [[ "$machine" == "arm64" ]]; then
         echo "This is an M1 Mac."
-        conda create -n "$env_name" -c conda-forge -y wget gsl hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests matplotlib python=3.12 openblas lapack liblapacke
+        conda create -n "$env_name" -c conda-forge -y wget gsl hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests rich matplotlib python=3.12 openblas lapack liblapacke
     else
         echo "This is not an M1 Mac."
-        conda create -n "$env_name" -c conda-forge -y clangxx_osx-64 clang_osx-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests matplotlib python=3.12
+        conda create -n "$env_name" -c conda-forge -y clangxx_osx-64 clang_osx-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests rich matplotlib python=3.12
     fi
 elif [[ "$OSTYPE" == "linux-gnu"* ]]; then
     echo "This system is Unix/Linux."
-    conda create -n "$env_name" -c conda-forge -y gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests matplotlib python=3.12
+    conda create -n "$env_name" -c conda-forge -y gcc_linux-64 gxx_linux-64 wget gsl lapack=3.6.1 hdf5 numpy Cython scipy tqdm jupyter ipython h5py requests rich matplotlib python=3.12
 else
     echo "Unsupported operating system."
 fi