Tests on using swapfile for conda solves

[Adam-D-Lewis]

name: env
channels:
  - conda-forge
dependencies:
  - python=3.10.8
  - ipykernel=6.21.0
  - ipywidgets==7.7.1
  - nebari-dask ==2023.1.1
  - python-graphviz=0.20.1
  - pyarrow=10.0.1
  - s3fs=2023.1.0
  - gcsfs=2023.1.0
  - numpy=1.23.5
  - numba=0.56.4
  - pandas=1.5.3
variables: {}

sudo dd if=/dev/zero of=/swapfile bs=1MB count=11264

I ran on n1 instances a high cpu and a high mem instance. Both had 2vcpu, 1 core, and 1.8 and 13GiB of RAM respectively.

I gave the high cpu instance an additional 11 GiB of RAM via swap to make both the memory usage about 13 GiB.

I ran time conda env create -f environement.yaml -y and it took 2m47s on highcpu and it took 2m27s on highmem (14% longer).

[Adam-D-Lewis]

I did the same with the following env and it took 8m4s on high cpu but it took 6m4s on high mem instance (33% longer)

name: env2
channels:
  - conda-forge
dependencies:
  - python=3.10
  - cdsdashboards-singleuser=0.6.3
  - cufflinks-py=0.17.3
  - dash=2.8.1
  - geopandas=0.12.2
  - geopy=2.3.0
  - geoviews=1.9.6
  - gunicorn=20.1.0
  - holoviews=1.15.4
  - ipykernel=6.21.2
  - ipywidgets=8.0.4
  - jupyter=1.0.0
  - jupyterlab=3.6.1
  - jupyter_bokeh=3.0.5
  - matplotlib=3.7.0
  - nebari-dask==2023.1.1
  - nodejs=18.12.1
  - numpy
  - openpyxl=3.1.1
  - pandas=1.5.3
  - panel=0.14.3
  - param=1.12.3
  - plotly=5.13.0
  - python-graphviz=0.20.1
  - rich=13.3.1
  - streamlit=1.9.0
  - sympy=1.11.1
  - voila=0.4.0
  - pip=23.0
  - pip:
      - streamlit-image-comparison==0.0.3
      - noaa-coops==0.2.1
      - dash_core_components==2.0.0
      - dash_html_components==2.0.0
variables: {}

[Adam-D-Lewis]

The disks used in this exercise were balanced persistent disks (https://cloud.google.com/compute/docs/disks#disk-types). I used miniconda to run the install.

[Adam-D-Lewis]

B/c docker containers rely on the host kernel for memory management, a docker container can't use swap unless there is already swap provisioned on the host or unless it is running as privileged and creates some on the host. In a quick search, it seems that Swap on Linux moved from alpha to beta on k8s with k8s version 1.28, but it doesn't seem to be supported on EKS though it is coming soon or GKE, but seems like it is possible on AKS. See Azure/AKS#1824 for more info.