Containers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anaconda\n",
    "\n",
    "* Called \"environments\"\n",
    "* Create a new environment\n",
    "    * conda create -n tensorflow python=3.5\n",
    "        * New environment called \"tensorflow\"\n",
    "        * Install python 3.5\n",
    "    * conda create -n python27 python=2.7 anaconda\n",
    "        * New environment called \"python27\"\n",
    "        * Install python 2.7\n",
    "        * Install anaconda\n",
    "* List environments\n",
    "    * conda env list\n",
    "        OR\n",
    "    * conda info --envs\n",
    "* Switch environments\n",
    "    * source activate python27\n",
    "        * switch to environment named \"python27\"\n",
    "    * source activate tensorflow\n",
    "        * switch to environment named \"tensorflow\"\n",
    "    * source deactivate\n",
    "        * switch back to default environment\n",
    "* Remove an environment\n",
    "    * conda remove --name env_name --all\n",
    "\n",
    "# Singularity (vagrant, virtualbox)\n",
    "\n",
    "* To stop vm\n",
    "    * exit out of singularity\n",
    "    * exit out of vagrant\n",
    "    * vagrant halt\n",
    "* To restart vm\n",
    "    * cd ~/singularity-vm\n",
    "    * vagrant up\n",
    "    * vagrant ssh\n",
    "    * singularity shell --bind /vagrant_data -e xinzhang8noaa-singularity-master.simg\n",
    "* Singularity preserves state after halting\n",
    "    * Files that you created before halting are still there after restarting\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Docker\n",
    "* To install docker on Mac\n",
    "    * brew cask install docker\n",
    "* To startup docker\n",
    "    * Launchpad --> Click on the whale\n",
    "    * A whale shows up in tool bar (near the wifi symbol)\n",
    "* At this point, can run 'docker' from command line\n",
    "* It has been said that Docker has serious security issues  \n",
    "    https://www.cimcor.com/blog/the-top-5-security-risks-in-docker-container-deployment\n",
    "    \n",
    "    * Several websites have advice on how to narrow the security hole  \n",
    "        https://www.sumologic.com/blog/security/securing-docker-containers/\n",
    "        \n",
    "        https://docs.docker.com/engine/security/security/\n",
    "        \n",
    "        https://sysdig.com/blog/7-docker-security-vulnerabilities/\n",
    "\n",
    "* Docker does not preserve state after exiting\n",
    "    * Files you created before exiting are gone after restarting\n",
    "* Repos\n",
    "    * JCSDA/docker_base\n",
    "        * Holds the generic installation (compilers, libraries such as mpi, netcdf, etc.)\n",
    "    * JCSDA/docker\n",
    "        * Holds the JEDI specific installation (eckit, fckit, ecbuild, odb, etc.)\n",
    "    * Dockerfile is key component\n",
    "        * Config for docker image\n",
    "        * Essentially a script that builds and installs all of the desired libraries\n",
    "            * Start of script shows images that are downloaded and built upon\n",
    "            * The docker_base Dockerfile downloads ubuntu\n",
    "            * The docker Dockerfile downloads ubuntu and jcsda/docker_base\n",
    "\n",
    "* Example commands\n",
    "    * docker image build -t jcsda/docker_base\n",
    "        * build the docker base image from the JCSDA/docker_base/Dockerfile\n",
    "    * docker image build -t jcsda/docker\n",
    "        * build the docker JEDI image from the JCSDA/docker/Dockerfile\n",
    "    * docker images -a\n",
    "        * show all images store locally\n",
    "    * docker rmi jcsda/docker\n",
    "        * remove the JCSDA docker image (and associated images)\n",
    "    * docker system prune\n",
    "        * remove all containers, images that are not being used\n",
    "    * docker run -it -v /Users/steveherbener/projects/docker-work/docker_data:/docker_data jcsda/docker\n",
    "        * run the container\n",
    "        * bind the host directory /Users/steveherbener/projects/docker-work/docker_data to the guest directory /docker_data\n",
    "\n",
    "* Common tasks\n",
    "    * Clean up\n",
    "        * docker system prune\n",
    "        * docker rmi jcsda/docker\n",
    "        * docker rmi jcsda/docker_base\n",
    "    * Build from scratch\n",
    "        * git clone https://github.com/JCSDA/docker_base\n",
    "        * cd docker_base\n",
    "        * docker image build -t jcsda/docker_base\n",
    "        * cd ..\n",
    "        * git clone https://github.com/JCSDA/docker\n",
    "        * cd docker\n",
    "        * docker image build -t jcsda/docker\n",
    "    * Run the docker container\n",
    "        * docker run -it -v /Users/steveherbener/projects/docker-work/docker_data:/docker_data jcsda/docker\n",
    "\n",
    "* Notes, 7/16/19\n",
    "    * To build with specific tag\n",
    "~~~~~~~~\n",
    "docker image build --no-cache -t jcsda/docker:<tag>\n",
    "~~~~~~~~\n",
    "\n",
    "    * After starting up image, need to switch user to \"jedi\"\n",
    "~~~~~~~~\n",
    "docker run <image>\n",
    ">> su - jedi\n",
    ">> set -o vi\n",
    "~~~~~~~~\n",
    "\n",
    "    * To pull image from docker hub\n",
    "~~~~~~~~\n",
    "docker pull jcsda/docker:eckit-1.1.0\n",
    "~~~~~~~~\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# AWS (Amazon Web Service)\n",
    "\n",
    "## Storing files\n",
    "\n",
    "* 2 ways to do this\n",
    "    * Personal Account\n",
    "        * Use script to download\n",
    "    * Public Account\n",
    "        * JCSDA\n",
    "        * Configured as a URL --> use wget to download\n",
    "\n",
    "## Notes, 8/5/19\n",
    "\n",
    "* Mark M set up a node that I can use for debugging\n",
    "    * Intel 17.0.1\n",
    "    * login: ubuntu@52.70.74.212\n",
    "    * Need to log into the AWS website to start and stop this node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes, 9/9/19\n",
    "\n",
    "* Mark M setup another node I can use for file conversion\n",
    "  * GNU or Intel\n",
    "  * login: ubuntu@3.226.35.84\n",
    "  * Log into the AWS website to start and stop this node\n",
    "\n",
    "* The node built on 8/5/19 is called: ioda-testing\n",
    "* The node built on 9/9/19 is called: ioda-file-conversion\n",
    "\n",
    "* Mark also created a new PEM file for me: srh-virginia\n",
    "\n",
    "* I've added aliases in my .bash_profile for logging into these nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}