C3S_climate-data-store.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/LogoLine_horizon_C3S.png' align='right' width='100%'></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Copernicus Climate Data Store (CDS) - Introduction and data access example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides you an introduction to the Copernicus Climate Data Store (CDS), an overview of the type of data that can be accessed and gives practical examples how to access and retrieve data from the CDS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outline\n",
    "* [1 - About](#about)\n",
    "* [2 - C3S data retrieval](#c3s_data_retrieval)\n",
    "  * [2.1 - Access data interactively via the CDS web interface](#access_manual)\n",
    "  * [2.2 - Access data programmatically with the CDS API](#access_programmatic)\n",
    "  * [2.3 - Further resources](#further_resources)\n",
    "* [3 - Example data request](#example_requests)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to access the notebook\n",
    "\n",
    "This tutorial is in the form of a [Jupyter notebook](https://jupyter.org/). You will not need to install any software for the training as there are a number of free cloud-based services to create, edit, run and export Jupyter notebooks such as this. Here are some suggestions (simply click on one of the links below to run the notebook):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Binder|Kaggle|Colab|NBViewer|\n",
    "|:-:|:-:|:-:|:-:|\n",
    "|[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/ecmwf-projects/copernicus-training/HEAD?urlpath=lab/tree/C3S_climate-data-store.ipynb)|[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/ecmwf-projects/copernicus-training/blob/master/C3S_climate-data-store.ipynb)|[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ecmwf-projects/copernicus-training/blob/master/C3S_climate-data-store)|[![NBViewer](https://raw.githubusercontent.com/ecmwf-projects/copernicus-training/master/img/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/ecmwf-projects/copernicus-training/blob/master/C3S_climate-data-store.ipynb)|\n",
    "|(Binder may take some time to load, so please be patient!)|(will need to login/register, and switch on the internet via *settings*)|(will need to run the command `!pip install cartopy` before importing the libraries)|(this will not run the notebook, only render it)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to run this notebook in your own environment, we suggest you install [Anaconda](https://docs.anaconda.com/anaconda/install/), which contains most of the libraries you will need. You will also need to install [Xarray](http://xarray.pydata.org/en/stable/) for working with multidimensional data in netcdf files, and the CDS API (`pip install cdsapi`) for downloading data programatically from the CDS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='about'></a>1. About"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Copernicus Climate Data Store (CDS)](https://cds.climate.copernicus.eu/) is the data access portal of the [Copernicus Climate Change Service (C3S)](https://climate.copernicus.eu/) and offers access to `data` and `applications` about the Earth's past, present and future climate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/cds_landing_page.png' align='left' width='80%'></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='c3s_data_retrieval'></a>2. Data retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to access data from the Copernicus Climate Data Store (CDS):\n",
    "* [interactively](#access_manual) via the CDS web interface, or\n",
    "* [programmatically](#access_programmatic) with the CDS API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='access_manual'></a>2.1 Access data interactively via the CDS web interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `CDS web interface` allows you to interactively `browse`, `select` and `download` datasets offered by the CDS. First, under [Datasets](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset), you can browse and select the dataset you are interested in. In a second step, you can then specify details of the data download form you wish to submit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filter and select a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a first step, you can `browse` and `filter` the dataset you are interested in. The [Datasets](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset) interface allows you either to select data based on different categories, e.g. `Product type`, `Variable domain`, `Spatial / Temporal coverage`, but also offers a free text search. The list of datasets in the main window allows you to select the dataset you are interested in. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/cds_web_interface_1.png' align='left' width='60%'></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you selected a dataset, you then get redirected to a data description section, which provides you an overview of the chosen dataset as well as the option to specify the dataset you would like to download and to also to submit the `\"Download data\" form`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the *Download data* form"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Data description` section (see 1) provides you an overview of the dataset, including a list of variables that are available. Under the tab `Download data`, the `\"Download data\" form` opens (see 2) which allows you to  interactively filter the dataset based on specific keywords, e.g.:\n",
    "* `Product type`\n",
    "* `Variable`\n",
    "* `Year / Month / Time`\n",
    "* `Geographical area`\n",
    "* `Format`\n",
    "\n",
    "At the end of the `\"Download data\" form`, you get three options: `Show API request`, `Show Toolbox request` and `Submit Form`. If you want to download the data interactively, the data requests will be executed as soon as you click on the `Submit Form` button. \n",
    "\n",
    "**Note:** You need to be logged into the ADS portal and you have to accept the Terms and Conditions of the dataset before being able to download data.\n",
    "\n",
    "\n",
    "You will need the `Show API request`, if you want to request data programmatically. See [Section 2.2](#access_programmatic) for further information. You will need the `Show Toolbox request`  if you want to request data via the CDS Toolbox. Learn more about the CDS Toolbox [here](https://cds.climate.copernicus.eu/cdsapp#!/toolbox).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>NOTE</b>: <br>\n",
    "    Under the tab <code>Your requests</code> in the main menu, you can monitor the status of your data requests.</div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/cds_data_description_download_form.png' align='left' width='60%'></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='access_programmatic'></a>2.2 Access data programmatically with the CDS API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Climate Data Store Application Program Interface (CDS API)` is a Python library which allows you to access data from the CDS programmatically. The library is available for both Python versions, Python 2.7.x and Python 3, but we recommend to use the library under Python 3. In order to use the CDS API, follow the steps below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the CDS API key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Self-register](https://cds.climate.copernicus.eu/#!/home) at the CDS registration page (if you do not have an account yet)\n",
    "* [Login](https://cds.climate.copernicus.eu/user/login) to the CDS portal and go to the [api-how-to page](https://cds.climate.copernicus.eu/api-how-to)\n",
    "* Copy the CDS API key displayed in the black terminal window in a file under `$HOME/.cdsapirc`\n",
    "\n",
    "**Note:** You find your CDS API key displayed in the black terminal box under the section `Install the CDS API key`. If you do not see a URL or key appear in the black terminal box, please refresh your browser tab. \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/cds_api_key.png' align='left' width='60%'></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below creates the file under your current working directory. Make sure to replace the `################` with your personal `CDS API key`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./.cdsapirc\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./.cdsapirc\n",
    "\n",
    "url: https://cds.climate.copernicus.eu/api/v2\n",
    "key: ##############################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alternative: Set CDS API credentials manually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can also define variables for `url` and `key`. These variables can then be set when you define the `cdsapi.Client(url=URL, key=KEY)` in your script (see next section for the different components of a data retrieval request).\n",
    "\n",
    "**Please note:** in the workflow notebooks, we will use this modality and set manually the CDS API key information for the data retrievals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = 'https://cds.climate.copernicus.eu/api/v2'\n",
    "KEY = '############################'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the CDS API client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to install the `CDS API client`. You can do this with the package management system `pip`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install cdsapi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the CDS API client for data access"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the `CDS API` is installed, it can be used to request data from the Climate Data Store."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, you see the principle of a `data retrieval` request. You always have to make sure to first import the `cdsapi` and define a `cdsapi.Client()` before you are able to execute an `API request`. You can use the [web interface](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset) to browse through the datasets. At the end of the `\"Download data\" form`, there is the option to choose `Show API request`. If you click this button, the `API request` appears (see example below), which you can copy paste into your coding workflow.\n",
    "\n",
    "<br>\n",
    "\n",
    "<div><img src='./img/cdsapi_request.png' align='left' width='30%'></img></div>\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>NOTE</b>: <br>\n",
    "    For data originating from ECMWF's Meteorological and Archival System (MARS), data is stored on a grid with longitudes from 0 to 360 degrees. It can be reprojected to a regular geographic latitude-longitude grid, by setting the keyword argument <code>area</code> and <code>grid</code>. Per default, data is retrieved in <code>GRIB</code>. If you wish to retrieve the data in <code>netCDF</code>, you have to specify it by using the keyword argument <code>format</code>.</div>\n",
    "    \n",
    "<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='further_resources'></a> 2.3 Further resources\n",
    "\n",
    "* [Climate Data Store (ADS) documentation](https://confluence.ecmwf.int/x/QhhsCg)\n",
    "* [Common Error Messages for CDS Requests](https://confluence.ecmwf.int/x/RKOpD)\n",
    "* [Climate Data Store (CDS) API Keywords](https://confluence.ecmwf.int/x/kjo1Cw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='example_requests'></a>3. Example data request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <a id='era5-land_hourly'></a>Example: **ERA5-Land hourly data from 1950 to present**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Data used in [113_c3s_climate_indices](./113_c3s_climate_indices.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CDS API name: `reanalysis-era5-land`\n",
    "\n",
    "> - Variable: `['10m_u_component_of_wind', '10m_v_component_of_wind','2m_temperature']`\n",
    "> - Year: `[1981 to 2020]`\n",
    "> - Month: `12`\n",
    "> - Day: `15`\n",
    "> - Time: `12:00`\n",
    "> - Area: `[60, -10, 35, 30]` # North, West, South, East\n",
    "> - Format: `netcdf`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** the request above makes use of the keyword `area`, which enable you to retrieve only a geographical subset. The bounding box information are set as follows: `[N, W, S, E]`. When this keyword is set, the data is automatically projected to a grid from [-180, 180].  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cdsapi\n",
    "c = cdsapi.Client()\n",
    "c.retrieve(\n",
    "    'reanalysis-era5-land',\n",
    "    {\n",
    "        'variable': [\n",
    "            '10m_u_component_of_wind', '10m_v_component_of_wind', '2m_temperature',\n",
    "        ],\n",
    "        'year': [\n",
    "            '1981', '1982', '1983',\n",
    "            '1984', '1985', '1986',\n",
    "            '1987', '1988', '1989',\n",
    "            '1990', '1991', '1992',\n",
    "            '1993', '1994', '1995',\n",
    "            '1996', '1997', '1998',\n",
    "            '1999', '2000', '2001',\n",
    "            '2002', '2003', '2004',\n",
    "            '2005', '2006', '2007',\n",
    "            '2008', '2009', '2010',\n",
    "            '2011', '2012', '2013',\n",
    "            '2014', '2015', '2016',\n",
    "            '2017', '2018', '2019',\n",
    "            '2020',\n",
    "        ],\n",
    "        'month': '12',\n",
    "        'day': '15',\n",
    "        'time': '12:00',\n",
    "        'format': 'netcdf',\n",
    "        'area': [\n",
    "            60, -10, 35,\n",
    "            30,\n",
    "        ],\n",
    "    },\n",
    "    './data/era5-land_eur_1981_2020.nc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p></p>\n",
    "<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}