Skip to content

Commit 7c1cb2d

Browse files
mike-ganglmike-gangl
authored
Add files via upload
1 parent 8ce6dd3 commit 7c1cb2d

File tree

1 file changed

+321
-0
lines changed

1 file changed

+321
-0
lines changed

‎jupyter-notebooks/tutorials/5_working_with_chirp_data.ipynb

Lines changed: 321 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,321 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "3be345ad-b008-45c6-b481-46888d2d2ebc",
6+
"metadata": {},
7+
"source": [
8+
"# Working with Data\n",
9+
"\n",
10+
"The intent of this tutorial is to help familiarize yourself with browsing for data that will be used along with an application to generate data by submitting a job. Job submission will be covered in the next tutorial. Run each cell in order (shift-enter). The notes will indicate when you need to edit code to customize things (e.g., to indicate a data collection)vs. being prompted by running the cell (e.g. for your username and password)."
11+
]
12+
},
13+
{
14+
"cell_type": "code",
15+
"execution_count": null,
16+
"id": "93e43227-c229-4ed4-a514-9293eda6c4f6",
17+
"metadata": {},
18+
"outputs": [],
19+
"source": [
20+
"import requests\n",
21+
"import getpass\n",
22+
"import json\n",
23+
"from IPython.display import JSON"
24+
]
25+
},
26+
{
27+
"cell_type": "markdown",
28+
"id": "b860ceff-8f05-4615-a78b-add89ae59567",
29+
"metadata": {},
30+
"source": [
31+
"First we need some pre-defined environment variables"
32+
]
33+
},
34+
{
35+
"cell_type": "code",
36+
"execution_count": null,
37+
"id": "484e55a1-2933-49e7-a684-c38ab269d1e8",
38+
"metadata": {},
39+
"outputs": [],
40+
"source": [
41+
"# This portion of the code is env specific for Dev, Test, Ops, etc. \n",
42+
"# define the environment as our test venue\n",
43+
"env = {\n",
44+
" \"clientId\":\"71894molftjtie4dvvkbjeard0\",\n",
45+
" \"url\":\"https://58nbcawrvb.execute-api.us-west-2.amazonaws.com/test/\"\n",
46+
" #\"url\":\"https://dxebrgu0bc9w7.cloudfront.net/\"\n",
47+
" }\n",
48+
"\n",
49+
"# The auth_json is template for authorizing with AWS Cognito for a token that can be used for calls to the data service.\n",
50+
"# For now this is just an empty data structure. You will be prompted for your username and password in a few steps.\n",
51+
"auth_json = '''{\n",
52+
" \"AuthParameters\" : {\n",
53+
" \"USERNAME\" : \"\",\n",
54+
" \"PASSWORD\" : \"\"\n",
55+
" },\n",
56+
" \"AuthFlow\" : \"USER_PASSWORD_AUTH\",\n",
57+
" \"ClientId\" : \"\"\n",
58+
" }'''"
59+
]
60+
},
61+
{
62+
"cell_type": "markdown",
63+
"id": "386c712e-307f-48e9-a1ba-f2bb3d378c4e",
64+
"metadata": {},
65+
"source": [
66+
"### Authentication Code\n",
67+
"\n",
68+
"The below method is a helper function for getting an access token for accessing Unity SDS services. You must pass the token along with any API requests in order to access the various Unity SDS services."
69+
]
70+
},
71+
{
72+
"cell_type": "code",
73+
"execution_count": null,
74+
"id": "6b9bbd23-1e4b-4286-959a-a00908a8ff7e",
75+
"metadata": {
76+
"tags": []
77+
},
78+
"outputs": [],
79+
"source": [
80+
"# This method is used for taking a username and password and client ID and fetching a cognito token\n",
81+
"def get_token(username, password, clientID):\n",
82+
" aj = json.loads(auth_json)\n",
83+
" aj['AuthParameters']['USERNAME'] = username\n",
84+
" aj['AuthParameters']['PASSWORD'] = password\n",
85+
" aj['ClientId'] =clientID \n",
86+
" token = None\n",
87+
" try:\n",
88+
" response = requests.post('https://cognito-idp.us-west-2.amazonaws.com', headers={\"Content-Type\":\"application/x-amz-json-1.1\", \"X-Amz-Target\":\"AWSCognitoIdentityProviderService.InitiateAuth\"}, json=aj)\n",
89+
" token = response.json()['AuthenticationResult']['AccessToken']\n",
90+
" except:\n",
91+
" print(\"Error, check username and password and try again.\")\n",
92+
" return token"
93+
]
94+
},
95+
{
96+
"cell_type": "markdown",
97+
"id": "5ea61770-5592-4e79-b6c3-8dadcb9301cb",
98+
"metadata": {
99+
"tags": []
100+
},
101+
"source": [
102+
"### Prompt for your Unity username and password\n",
103+
"\n",
104+
"These are required to get the token (described above) to connect to the data services."
105+
]
106+
},
107+
{
108+
"cell_type": "code",
109+
"execution_count": null,
110+
"id": "1f586ee1-003f-40eb-9ae5-3770b4579ead",
111+
"metadata": {
112+
"tags": []
113+
},
114+
"outputs": [],
115+
"source": [
116+
"print(\"Please enter your username...\")\n",
117+
"user_name = input()\n",
118+
"\n",
119+
"print(\"Please enter your password...\")\n",
120+
"password = getpass.getpass()"
121+
]
122+
},
123+
{
124+
"cell_type": "code",
125+
"execution_count": null,
126+
"id": "be1b7bcd-eb10-44aa-9940-d92b7bec9817",
127+
"metadata": {},
128+
"outputs": [],
129+
"source": [
130+
"token = get_token(user_name, password, env['clientId'])\n",
131+
"\n",
132+
"if(token):\n",
133+
" print(\"Token received.\")"
134+
]
135+
},
136+
{
137+
"cell_type": "markdown",
138+
"id": "1a128652-a73b-4b9c-93ce-5f4147d73c32",
139+
"metadata": {},
140+
"source": [
141+
"## List Available Data Collections in the Unity System\n",
142+
"\n",
143+
"Data is organized into Collections. Any particular data file will be in at least one Collection."
144+
]
145+
},
146+
{
147+
"cell_type": "code",
148+
"execution_count": null,
149+
"id": "a8e1ffbd-ee53-497b-8c89-ce9929263e4d",
150+
"metadata": {},
151+
"outputs": [],
152+
"source": [
153+
"# The DAPA-request endpoint to retrieve collections is the base URL plus the following:\n",
154+
"url = env['url'] + \"am-uds-dapa/collections\"\n",
155+
"\n",
156+
"# Make a GET request at the URL you have constructed, using your access token\n",
157+
"response = requests.get(url, headers={\"Authorization\": \"Bearer \" + token})\n",
158+
"print(response)\n",
159+
"\n",
160+
"print (\"Data Collections at \" + url)\n",
161+
"# To see raw JSON of the API response, uncomment this line:\n",
162+
"#print(json.dumps(response.json()))\n",
163+
"features = response.json()['features']\n",
164+
"for data_set in features:\n",
165+
" print(data_set['id'])\n",
166+
"\n",
167+
"print(\"\\nFull JSON response object:\")\n",
168+
"JSON(response.json())\n"
169+
]
170+
},
171+
{
172+
"cell_type": "code",
173+
"execution_count": null,
174+
"id": "1bb34cc9-eaae-4efd-82c0-6f33043b6207",
175+
"metadata": {},
176+
"outputs": [],
177+
"source": [
178+
"data_set = \"urn:nasa:unity:ssips:TEST1:CHRP_16_DAY_REBIN___1\"\n",
179+
"url = env['url'] + \"am-uds-dapa/collections/\"+data_set+\"/items\"\n",
180+
"\n",
181+
"params = []\n",
182+
"#params.append((\"limit\", 20))\n",
183+
"\n",
184+
"response = requests.get(url, headers={\"Authorization\": \"Bearer \" + token}, params=params)\n",
185+
"\n",
186+
"print(f\"Endpoint: \"+url)\n",
187+
"#print(f\"Total number of files: {response.json()['numberMatched']}\")\n",
188+
"print(\"File IDs, titles, and hrefs in Collection \" + data_set + \"\\n\")\n",
189+
"\n",
190+
"features = response.json()['features']\n",
191+
"\n",
192+
"for data_file in features: {\n",
193+
" print(\"For \"+ data_file['id']),\n",
194+
" print(\"File:\\t\\t\"+data_file['assets']['data']['href']),\n",
195+
" print(\"Metadata:\\t\"+data_file['assets']['metadata__data']['href']),\n",
196+
" print(\"\")\n",
197+
"}\n",
198+
"\n",
199+
"\n",
200+
"print(\"Full JSON response object:\")\n",
201+
"JSON(response.json())\n"
202+
]
203+
},
204+
{
205+
"cell_type": "markdown",
206+
"id": "42bc4d07-d8de-4298-9d33-6299ae41a024",
207+
"metadata": {},
208+
"source": [
209+
"## Filter the results above by time\n",
210+
"\n",
211+
"The standards-based API used by the Unity SDS Data Store, DAPA, has a variety of filtering options. Currently we have implemented a time-based filter. See more about the Data Access and Processing API at: https://docs.ogc.org/per/20-025r1.html#_dapa_overview\n",
212+
"\n",
213+
"This cell will filter the full list of files in the Collection with ID = data_set by a start and end time defined by the datetime parameter."
214+
]
215+
},
216+
{
217+
"cell_type": "code",
218+
"execution_count": null,
219+
"id": "70a4ab9e-8693-4948-bc2a-cce1e09d1cab",
220+
"metadata": {},
221+
"outputs": [],
222+
"source": [
223+
"url = env['url'] + \"am-uds-dapa/collections/\"+data_set+\"/items\"\n",
224+
"# the datetime,limit, and offset are included due to a current bug in the API Gatway setting these values to 'none'.\n",
225+
"# Example date/time params\n",
226+
"\n",
227+
"params = []\n",
228+
"#add a datetime to your request\n",
229+
"# Note, this will not currently work as the CHIRP Applicaiton package does not specify the appropriate start or stop dates for the data\n",
230+
"params.append((\"datetime\", \"2023-11-02T00:00:00Z/2023-11-09T02:31:12Z\"))\n",
231+
"\n",
232+
"# limit - how many results to return in a single request\n",
233+
"#params.append((\"limit\", 10))\n",
234+
"\n",
235+
"response = requests.get(url, headers={\"Authorization\": \"Bearer \" + token}, params=params)\n",
236+
"\n",
237+
"print(f\"Total number of files: {response.json()['numberMatched']}\")\n",
238+
"print(\"File IDs, datetimes, and hrefs in Collection \" + data_set + \"\\n\")\n",
239+
"\n",
240+
"features = response.json()['features']\n",
241+
"\n",
242+
"for data_file in features: {\n",
243+
" print(data_file['id']),\n",
244+
" print(data_file['properties']['created']),\n",
245+
" print(data_file['assets']['metadata__data']['href']),\n",
246+
" print(data_file['assets']['data']['href']),\n",
247+
" print(\"\")\n",
248+
"}"
249+
]
250+
},
251+
{
252+
"cell_type": "markdown",
253+
"id": "d79e544a",
254+
"metadata": {},
255+
"source": [
256+
"## Credential-less data download\n",
257+
"\n",
258+
"When accessing data stores within the same venue, you'll be able to download data from S3 without credentials. \n",
259+
"\n",
260+
"**Note**, the following libraries are needed for this, and the below command can be run in a jupyter-terminal to install them:\n",
261+
"\n",
262+
"```\n",
263+
"conda install xarray netcdf4 hdf5 boto3 matplotlib\n",
264+
"```\n"
265+
]
266+
},
267+
{
268+
"cell_type": "code",
269+
"execution_count": null,
270+
"id": "620a4e4e",
271+
"metadata": {},
272+
"outputs": [],
273+
"source": [
274+
"import sys\n",
275+
"!{sys.executable} -m pip install boto3\n",
276+
"import boto3"
277+
]
278+
},
279+
{
280+
"cell_type": "code",
281+
"execution_count": null,
282+
"id": "5c6a2837-fb5a-4467-94e5-a85f8659506e",
283+
"metadata": {},
284+
"outputs": [],
285+
"source": [
286+
"s3 = boto3.client('s3')\n",
287+
"#s3://ssips-test-ds-storage-reproc/urn:nasa:unity:ssips:TEST1:CHRP_16_DAY_REBIN___1/urn:nasa:unity:ssips:TEST1:CHRP_16_DAY_REBIN___1:SNDR_tile_2016_s320_N16p50_E120p00_L1_AQ_v1_D_2311021698943223.nc/SNDR_tile_2016_s320_N16p50_E120p00_L1_AQ_v1_D_2311021698943223.nc\n",
288+
"s3.download_file('ssips-test-ds-storage-reproc', 'urn:nasa:unity:ssips:TEST1:CHRP_16_DAY_REBIN___1/urn:nasa:unity:ssips:TEST1:CHRP_16_DAY_REBIN___1:SNDR_tile_2016_s320_N16p50_E120p00_L1_AQ_v1_D_2311021698943223.nc/SNDR_tile_2016_s320_N16p50_E120p00_L1_AQ_v1_D_2311021698943223.nc', 'test_file11.nc')"
289+
]
290+
},
291+
{
292+
"cell_type": "markdown",
293+
"id": "e81ee1e1-c485-45c2-9074-29c5c3d935fc",
294+
"metadata": {},
295+
"source": [
296+
"The file now should appear in the directory tree to the left in jupyter"
297+
]
298+
}
299+
],
300+
"metadata": {
301+
"kernelspec": {
302+
"display_name": "Python 3 (ipykernel)",
303+
"language": "python",
304+
"name": "python3"
305+
},
306+
"language_info": {
307+
"codemirror_mode": {
308+
"name": "ipython",
309+
"version": 3
310+
},
311+
"file_extension": ".py",
312+
"mimetype": "text/x-python",
313+
"name": "python",
314+
"nbconvert_exporter": "python",
315+
"pygments_lexer": "ipython3",
316+
"version": "3.11.5"
317+
}
318+
},
319+
"nbformat": 4,
320+
"nbformat_minor": 5
321+
}

0 commit comments

Comments
 (0)