Skip to content

Commit

Permalink
Merge pull request #211 from Ouranosinc/revert-change-for-new-broken-…
Browse files Browse the repository at this point in the history 
…magpie-to-fix-jenkins-in-production

Revert "Merge pull request #209 from Ouranosinc/magpie-unauthorized-thredds"

This reverts commit 6ab2ae8, reversing changes made to 947ea40.

Production has not been upgraded to the broken Magpie (PR bird-house/birdhouse-deploy#107) so this revert is so that Jenkins pass again on production.

Jenkins build: https://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/master/242/console

The `finch-usage.ipynb` failure is due to bird-house/finch#161 (comment), a new Finch will be deployed to prod soon (bird-house/birdhouse-deploy#138).  Otherwise, with this revert, production will be green.

The original #209 is back in branch https://github.com/Ouranosinc/pavics-sdi/tree/restore-change-for-new-broken-magpie-so-we-can-test-magpie-fix so we can test new Magpie with fix.

Matching PAVICS stack revert: bird-house/birdhouse-deploy#137
  • Loading branch information
@tlvu
tlvu authored Mar 26, 2021
2 parents 6ab2ae8 + 23ef2de commit 531d2d9
Showing 1 changed file with 29 additions and 148 deletions.
177 changes: 29 additions & 148 deletions docs/source/notebooks/pavics_thredds.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,42 +7,9 @@
"# Accessing PAVICS THREDDS Server\n",
"\n",
"\n",
"The THREDDS data storing netCDF file on PAVICS has some public and private directories. Data from public directories can be accessed anonymously, while data from private directories require authentication. This notebook shows how to access public and private data on the THREDDS server.\n",
"The THREDDS data storing netCDF file on PAVICS has some public and private directories. Data from public directories can be accessed anonymously, while data from private directories require authentication. This notebook shows how to access public and private data on the THREDDS server. \n",
"\n",
"The PAVICS THREDDS server has a `testdata/` folder, in which we store test datasets to validate process requests. Within that directory is a `secure/` folder whose file access requires authentication (to be done)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"THREDDS URL: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds\n"
]
}
],
"source": [
"# NBVAL_IGNORE_OUTPUT\n",
"\n",
"# define some useful variables for following steps\n",
"import os\n",
"PAVICS_HOST = os.getenv(\"PAVICS_HOST\", \"pavics.ouranos.ca\")\n",
"THREDDS_URL = \"https://{}/twitcher/ows/proxy/thredds\".format(PAVICS_HOST)\n",
"\n",
"assert PAVICS_HOST != \"\", \"Invalid PAVICS HOST value.\"\n",
"print(\"THREDDS URL:\", THREDDS_URL)"
"The PAVICS THREDDS server has a `testdata/` folder, in which we store test datasets to validate process requests. Within that directory is a `secure/` folder whose file access requires authentication (to be done). "
]
},
{
Expand All @@ -54,7 +21,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 1,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -148,7 +115,7 @@
" DODS_EXTRA.Unlimited_Dimension: time"
]
},
"execution_count": 6,
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -157,120 +124,41 @@
"import xarray as xr\n",
"xr.set_options(display_style=\"text\") # comment out for html style, text style simpler for automated testing\n",
"\n",
"PUBLIC_URL = f\"{THREDDS_URL}/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc\"\n",
"ds = xr.open_dataset(PUBLIC_URL)\n",
"url = \"https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc\"\n",
"ds = xr.open_dataset(url)\n",
"ds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's do the same with a secured link."
"Now let's do the same with a secured link. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Unauthorized was raised as expected.\n"
]
}
],
"outputs": [],
"source": [
"from webob.exc import HTTPError\n",
"secured_url = \"https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/secure/tasmax_Amon_MPI-ESM-MR_rcp45_r2i1p1_200601-200612.nc\"\n",
"\n",
"SECURED_URL = f\"{THREDDS_URL}/dodsC/birdhouse/testdata/secure/tasmax_Amon_MPI-ESM-MR_rcp45_r2i1p1_200601-200612.nc\"\n",
"try:\n",
" ds = xr.open_dataset(SECURED_URL, decode_cf=False)\n",
"# depending on 'xarray' version, differnt errors are raised when failing authentication according on how they handle it\n",
"except OSError as exc: # xarray < 0.17\n",
" assert \"Authorization failure\" in str(exc)\n",
"except HTTPError as exc: # xarray >= 0.17\n",
" # note: raised error is 500 with 'message' Unauthorized instead of directly raising HTTPUnauthorized\n",
" assert \"401 Unauthorized\" in str(exc)\n",
"else:\n",
" raise RuntimeError(\"Expected unauthorized response, but dataset open operation did not raise!\")\n",
"print(\"Unauthorized was raised as expected.\")"
"# This should fail but doesn't at the moment. \n",
"ds = xr.open_dataset(secured_url, decode_cf=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To open a secured link, we need to open a session with `Authentication`.\n",
"Using wrong `Authentication` credentials will not work. They will raise immediately when failing login procedure.\n",
"Using valid credentials will instead succeed login, but will raise a forbidden response when attempting to retrieve\n",
"the data. Either way, user must be logged in and have appropriate access to fulfill `Authorization` requirements\n",
"of the resource.\n",
"\n",
"Let's see the result when credentials are invalid."
"To open a secured link, we need to open a session. We've created a `authtest` user to facilitate testing. "
]
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Access with invalid credentials was not permitted as expected.\n"
]
}
],
"source": [
"import requests\n",
"from requests_magpie import MagpieAuth, MagpieAuthenticationError\n",
"\n",
"BAD_USR = \"an-invalid-user\"\n",
"BAD_PWD = \"or-bad-password\"\n",
"\n",
"try:\n",
" with requests.session() as session:\n",
" session.auth = MagpieAuth(f\"https://{PAVICS_HOST}/magpie\", BAD_USR, BAD_PWD)\n",
" xr.open_dataset(SECURED_URL, decode_cf=False) # Attributes are problematic with this file.\n",
"# specific error depends on what raises (unauthorized, forbidden, login failure) and 'xarray' version\n",
"except (OSError, HTTPError, MagpieAuthenticationError) as exc:\n",
" print(\"Access with invalid credentials was not permitted as expected.\")\n",
"else:\n",
" raise RuntimeError(\"Expected authentication failure response, but login operation did not raise!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"As we can see, the server identified that credentials were provided, but they were incorrect and could not log in.\n",
"Similar result would happen if login succeeded, but user was forbidden access due to insufficient permissions.\n",
"\n",
"We've created an `authtest` user in advance that has access to the `secure` contents to facilitate testing.\n",
"\n",
"Let's use it now to obtain the secured resource.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -363,34 +251,27 @@
" DODS_EXTRA.Unlimited_Dimension: time"
]
},
"execution_count": 14,
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"AUTH_USR = os.getenv(\"TEST_MAGPIE_ADMIN_USERNAME\", \"authtest\")\n",
"AUTH_PWD = os.getenv(\"TEST_MAGPIE_ADMIN_PASSWORD\", \"authtest1234\")\n",
"import requests\n",
"from requests_magpie import MagpieAuth\n",
"\n",
"secured_url = \"https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/secure/tasmax_Amon_MPI-ESM-MR_rcp45_r2i1p1_200601-200612.nc\"\n",
"auth = MagpieAuth(\"https://pavics.ouranos.ca/magpie\", \"authtest\", \"authtest1234\")\n",
"\n",
"# Open session\n",
"with requests.Session() as session:\n",
" session.auth = MagpieAuth(f\"https://{PAVICS_HOST}/magpie\", AUTH_USR, AUTH_PWD)\n",
" # Open a PyDAP data store and pass it to xarray\n",
" store = xr.backends.PydapDataStore.open(SECURED_URL, session=session)\n",
" ds = xr.open_dataset(store, decode_cf=False) # Attributes are problematic with this file.\n",
"session = requests.Session()\n",
"session.auth = auth\n",
"\n",
"# Open a Pydap data store and pass it to xarray\n",
"store = xr.backends.PydapDataStore.open(secured_url, session=session)\n",
"ds = xr.open_dataset(store, decode_cf=False) # Attributes are problematic with this file. \n",
"ds"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Successful listing of the above data means the user was granted access for this reference.\n"
]
}
],
"metadata": {
Expand All @@ -409,9 +290,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
"nbformat_minor": 2
}

0 comments on commit 531d2d9

Please sign in to comment.