Skip to content

feat: elasticsearch branch #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 344 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
344 commits
Select commit Hold shift + click to select a range
6d63d2e
Merge pull request #65 from unity-sds/develop
wphyojpl Jul 28, 2022
32024b2
chore: update error description
wphyojpl Aug 5, 2022
e4e8709
feat: add size, prev, next url for granules
wphyojpl Aug 5, 2022
ece0d2b
feat: add stats for collection + size. fix: check if queryString is NULL
wphyojpl Aug 8, 2022
4b8b2a6
fix: encap code in try-catch to avoid `internal server error`
wphyojpl Aug 8, 2022
13333c8
Updated README.md template (#69)
ngachung Aug 11, 2022
8c6ae4b
fix: add err msg when next & prev url does not generate + update make…
wphyojpl Aug 15, 2022
1fe341a
feat: STAC style pagination (#70)
wphyojpl Aug 23, 2022
e958bd5
chore: merge from remote
wphyojpl Aug 23, 2022
ca7e8cc
breaking: pystac validation test + upload / download docker unit test…
wphyojpl Aug 24, 2022
80742de
chore: Open api yaml for documentation (#72)
wphyojpl Aug 24, 2022
ad2e47f
chore: add unit test for param store cognito login (#74)
wphyojpl Aug 31, 2022
a8ffe18
chore: move the unit test (#75)
wphyojpl Aug 31, 2022
70fb836
Update setup.py
wphyojpl Sep 6, 2022
7465f44
fix: GitHub.action.update (#76)
wphyojpl Sep 6, 2022
e20b34b
Merge branch 'main' of github.com:unity-sds/unity-data-services into …
wphyojpl Sep 6, 2022
d92d2d4
Merge pull request #77 from unity-sds/develop.2022.09.06_01
wphyojpl Sep 6, 2022
5fec024
/version 1.7.0 release
ngachung Sep 6, 2022
0bd3c27
Merge pull request #78 from unity-sds/develop
ngachung Sep 6, 2022
a52482d
fix: level.1.a.missing.filename (#80)
wphyojpl Sep 7, 2022
3063c1d
Merge pull request #81 from unity-sds/main
wphyojpl Sep 7, 2022
ef56f1a
fix: not honoring offset and limit in Collection query (#82)
wphyojpl Sep 8, 2022
c44e9af
Merge pull request #83 from unity-sds/main
wphyojpl Sep 13, 2022
0cb4c78
feat: Collection Creation endpoint with DAPA format (#79)
wphyojpl Sep 21, 2022
aa5563c
fix: check pathParameters is None (#89)
wphyojpl Sep 27, 2022
bee5360
chore: bump version + update changelog (#90)
wphyojpl Sep 27, 2022
fd24db1
feat: add elasticsearch code
wphyojpl Sep 29, 2022
8089aa3
fix: Completed.granules.only + validate start + end time for STAC (#93)
wphyojpl Oct 14, 2022
83b7601
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Oct 14, 2022
8fc6442
feat: add cognito class + sample test case
wphyojpl Oct 14, 2022
12a9ce9
feat: adding authorization class (in-progress)
wphyojpl Oct 15, 2022
0c0bccf
feat: add es in terraform
wphyojpl Oct 20, 2022
56872bf
fix: add version to requests
wphyojpl Oct 20, 2022
7f7f863
fix: terraform aws 3.xx does not have opensearch
wphyojpl Oct 21, 2022
58da2aa
fix: throughput is not supported yet
wphyojpl Oct 21, 2022
d0e9636
fix: switch engine_version to elasticsearch_version
wphyojpl Oct 21, 2022
eca565a
fix: shorten domain name
wphyojpl Oct 21, 2022
ea5b735
fix: cannot use the name while creating it
wphyojpl Oct 21, 2022
3ed03de
Added DAPA lambdas invoke arn values to parameter store
ngachung Oct 24, 2022
8442aea
chore: authorization prep work
wphyojpl Oct 25, 2022
6160562
chore: prep for authorization
wphyojpl Oct 25, 2022
214a8b4
feat: add resource in auth table
wphyojpl Oct 31, 2022
8a8c413
Updated to store function name instead.
ngachung Oct 31, 2022
47187df
Updated to store function name instead.
ngachung Oct 31, 2022
584bb06
feat: add method to get user, action & resource from lambda-event
wphyojpl Oct 31, 2022
664d332
feat: add authorized projects to cumulus granules
wphyojpl Oct 31, 2022
37f2850
fix: add missing class
wphyojpl Oct 31, 2022
f32c7cb
chore: prep for authorization
wphyojpl Oct 31, 2022
8f31973
Merge pull request #96 from unity-sds/feature/dapa-param-store
ngachung Oct 31, 2022
3d7741c
fix: convert project to tenant
wphyojpl Nov 2, 2022
5903dbf
Merge remote-tracking branch 'origin/develop' into elasticsearch.lib
ngachung Nov 7, 2022
94fa496
Opensearch domain requires one subnet
ngachung Nov 7, 2022
190ba63
feat: update auth index according to slides
wphyojpl Nov 9, 2022
666de30
fix: add validation step
wphyojpl Nov 10, 2022
e6971a1
fix: updating auth for granules + collection listing
wphyojpl Nov 10, 2022
cf2de18
fix: get ldap_group from authorization header
wphyojpl Nov 15, 2022
3e0213e
fix: update how to get authorized collection id
wphyojpl Nov 15, 2022
6659c22
fix: accept provider from ENV or optionally from user call
wphyojpl Nov 15, 2022
4a5bd75
chore: update test case
wphyojpl Nov 15, 2022
924704c
fix: add provider in rules
wphyojpl Nov 16, 2022
d899333
chore: update changelog
wphyojpl Nov 16, 2022
50ff352
Updated CHANGELOG.md
ngachung Nov 16, 2022
f5bed59
Merge pull request #99 from unity-sds/missing-provider
ngachung Nov 16, 2022
b5978e2
fix: status=completed is only for granules, not for collections
wphyojpl Nov 16, 2022
587385d
Merge pull request #100 from unity-sds/status=completed-bug
ngachung Nov 16, 2022
6ba2ba6
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Nov 21, 2022
8d9c9d4
fix: list collections based on authorized collections
wphyojpl Nov 21, 2022
3802d4f
fix: some typos when changing from username to ldap-groups
wphyojpl Nov 21, 2022
e183944
feat: add colleciton to elasticsearch
wphyojpl Nov 29, 2022
1e4c8de
feat: add test case + fallback when elasticsearch add fails
wphyojpl Nov 29, 2022
191237b
feat: return upload result (#103)
wphyojpl Dec 19, 2022
9089b93
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Dec 19, 2022
8e15be4
feat: lambda to setup ES
wphyojpl Dec 20, 2022
e427432
feat: use python, not json as they are not packaged
wphyojpl Dec 20, 2022
4d0d8d1
feat: added Updated time in collection & item STAC dictionaries (#104)
wphyojpl Jan 5, 2023
39c034a
chore: merged from develop
wphyojpl Jan 5, 2023
1355475
feat: add wrapper scripts + entry points for admin functions
wphyojpl Jan 11, 2023
5682c92
feat: add admin A&A
wphyojpl Jan 11, 2023
a3eeb7a
fix: missing param in uds_auth_factory
wphyojpl Jan 20, 2023
7200e9d
fix: admin function bug fixes after testing it manually
wphyojpl Jan 21, 2023
4c7934b
chore: update collection creation to test with A&A
wphyojpl Jan 23, 2023
cefe22c
chore: remove cognito_id from code as it will be a part of input request
wphyojpl Jan 23, 2023
f58d604
fix: add es_url to terraform lambda functions
wphyojpl Jan 23, 2023
73c923e
fix: update dockerfile base image to 3.9 (#112)
wphyojpl Jan 23, 2023
37ef542
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jan 23, 2023
03d70a9
chore: add integration test for admin function
wphyojpl Jan 24, 2023
4d97d63
fix: bugfixes while testing it in mcp-dev
wphyojpl Jan 25, 2023
fe306eb
fix: bugfixes during integration testing in mcp-dev part(2)
wphyojpl Jan 25, 2023
8a69ab3
feat: update download granule test
wphyojpl Jan 25, 2023
5aab765
feat: add ssm for new lambdas
wphyojpl Jan 31, 2023
cdcc797
fix: wrong commit in previous commit
wphyojpl Jan 31, 2023
571e95e
fix: new stage-in stage-out
wphyojpl Feb 3, 2023
e1f6f6f
fix: remove old download scripts
wphyojpl Feb 3, 2023
ebfba3a
fix: docker scripts for split download + search
wphyojpl Feb 3, 2023
2950499
chore: update version + change log
wphyojpl Feb 3, 2023
d470a66
feat: add upload step + test
wphyojpl Feb 3, 2023
9033495
feat: add catalog stage
wphyojpl Feb 6, 2023
57af882
chore: remove old class
wphyojpl Feb 6, 2023
e91d48e
chore: update sample docker-compose files
wphyojpl Feb 6, 2023
c47d2f0
fix: print method output in docker + tweak docker compose files
wphyojpl Feb 6, 2023
7bb418b
chore: update changelog
wphyojpl Feb 6, 2023
b26b227
feat: add CMR granules search which returns STAC
wphyojpl Mar 9, 2023
a4c9d47
Merge pull request #120 from unity-sds/split-stage-in-out
ngachung Mar 15, 2023
5cc8c98
breaking: configurable file postfixes for PDS metadata extraction (#124)
wphyojpl Apr 5, 2023
47eed19
feat: reduce pystac length by keeping only data asset (#126)
wphyojpl Apr 11, 2023
1c27f66
feat: granules query pagination (#131)
wphyojpl Apr 11, 2023
6a6ecad
feat: add EDL token class
wphyojpl Apr 12, 2023
e1ed5ed
feat: adding DAAC download in docker stage in (in-progress)
wphyojpl Apr 12, 2023
a64e821
feat: add option to parse downloading stac from file (#134)
wphyojpl Apr 12, 2023
3375bce
fix: uncomment temporal in CMR granules search (#136)
wphyojpl Apr 13, 2023
be24023
chore: merge from develop
wphyojpl Apr 17, 2023
da2abe5
fix: some typo in download daac
wphyojpl Apr 17, 2023
1aa62b6
chore: add some TODOs
wphyojpl Apr 17, 2023
9a756ff
fix: update client id
wphyojpl Apr 17, 2023
83efad3
fix: update client id
wphyojpl Apr 17, 2023
a0ae495
chore: update changelog
wphyojpl Apr 17, 2023
2e2fee1
Merge pull request #132 from unity-sds/daac-download
ngachung Apr 21, 2023
b67aa67
feat: Checkout stage with STAC catalog json (#138)
wphyojpl Apr 24, 2023
512133e
feat: daac download from stac json file
wphyojpl Apr 24, 2023
7795cd1
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Apr 24, 2023
81947a2
chore: update version
wphyojpl Apr 24, 2023
e7686cd
Merge pull request #142 from unity-sds/daac-download-stac-file
ngachung Apr 24, 2023
0a814a0
fix: downloaded stac to return local absolute path
wphyojpl Apr 24, 2023
a92a6fc
chore: update PR link
wphyojpl Apr 24, 2023
abc7a31
Merge pull request #144 from unity-sds/stac-local-url
ngachung Apr 25, 2023
bc051df
feat: add metadata extraction from stac + refactor
wphyojpl Apr 25, 2023
ff71035
chore: refactor
wphyojpl Apr 25, 2023
f21745c
chore: refactor
wphyojpl Apr 25, 2023
4f709f7
fix: error during refactor
wphyojpl Apr 25, 2023
13a8879
chore: still refactoring
wphyojpl Apr 25, 2023
688a0eb
fix: update code for stac lambda
wphyojpl Apr 25, 2023
56d2332
feat: add terraform
wphyojpl Apr 25, 2023
46c28de
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Apr 25, 2023
0e1b905
chore: update version
wphyojpl Apr 25, 2023
7bf7e7c
chore: merge from develop
wphyojpl Apr 26, 2023
a517989
Merge pull request #146 from unity-sds/stac-metadata-extraction
ngachung Apr 28, 2023
9baeb78
feat: optional output to file
wphyojpl May 3, 2023
e53d7f7
fix: add unit test + create folder if needed
wphyojpl May 4, 2023
018b85d
fix: tweak it a bit to avoid writing it twice in case some steps are …
wphyojpl May 4, 2023
66498d6
chore: update readme
wphyojpl May 4, 2023
854f8e8
fix: use cas structure to generate metadata for stac (#148)
wphyojpl May 5, 2023
b582adc
Merge branch 'develop' into output-to-file
wphyojpl May 5, 2023
65f6605
chore: merge from develop
wphyojpl May 10, 2023
1c63675
fix: update requests version for git error
wphyojpl May 10, 2023
6dd0d61
fix: reshuffle to fix version mismatch
wphyojpl May 10, 2023
6a6d7f7
fixing urllib3 version
wphyojpl May 10, 2023
55ddcf5
Merge pull request #149 from unity-sds/output-to-file
ngachung May 15, 2023
54ebc7c
fix: allow catalog stage input from file
wphyojpl May 22, 2023
639084d
chore: update changelog
wphyojpl May 22, 2023
915406c
fix: remove local var. using parent var
wphyojpl May 22, 2023
a79f1ec
Merge pull request #152 from unity-sds/catalog-stage-from-file
ngachung May 23, 2023
a122a09
fix: production datetime not in +00:00 format
wphyojpl May 23, 2023
f836c0f
chore: update version + readme
wphyojpl May 23, 2023
fb7465b
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl May 23, 2023
7a1d20f
fix: add tags to opensearch
wphyojpl May 23, 2023
038b94c
Merge pull request #154 from unity-sds/stac-metadata-datetime
ngachung May 24, 2023
51854c5
fix: add tags in tf
wphyojpl Jun 6, 2023
cd9dcd9
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jun 6, 2023
9f9fe68
feat: add public http download class
wphyojpl Jun 12, 2023
90193dd
breaking: search return entire feature-collection
wphyojpl Jun 12, 2023
c10f331
breaking: daac downloader accepts only features collection
wphyojpl Jun 13, 2023
a4bb017
chore: update daac testcase
wphyojpl Jun 13, 2023
6c26b0b
feat: refactor download classes to use inheritance
wphyojpl Jun 13, 2023
837a22f
fix: http downloader had a typo bug
wphyojpl Jun 13, 2023
6526a4b
chore: update readme
wphyojpl Jun 13, 2023
7853a92
chore: update readme
wphyojpl Jun 13, 2023
52e91d5
Merge pull request #161 from unity-sds/http.download
ngachung Jun 13, 2023
dbe6445
feat: new upload handler for relative paths + user granule-id from th…
wphyojpl Jun 19, 2023
523527d
feat: output to be an item-collection
wphyojpl Jun 19, 2023
f5dd731
feat: update catalog to accept item collection
wphyojpl Jun 20, 2023
169fca9
chore: update version
wphyojpl Jun 20, 2023
78109a7
breaking: removing old upload styles + default it to new style if mis…
wphyojpl Jun 21, 2023
8652635
chore: delete old classes
wphyojpl Jun 21, 2023
41f88f0
chore: update version since it is breaking change now
wphyojpl Jun 21, 2023
b79d414
Merge pull request #163 from unity-sds/upload.relative.path
ngachung Jun 21, 2023
09a1569
fix: convert all outputs into json str
wphyojpl Jun 21, 2023
5b93b87
chore: update version
wphyojpl Jun 21, 2023
c601b40
chore: update PR number
wphyojpl Jun 21, 2023
4badceb
Merge pull request #165 from unity-sds/output-in-json-str
ngachung Jun 22, 2023
1a0ade8
feat: added filter keyword in granules endpoint + repeatedly checking…
wphyojpl Jul 5, 2023
5883643
feat: parallelize download (#169)
wphyojpl Jul 5, 2023
efe0ab1
fix: retry 5 times + wait 45 sec for DAAC
wphyojpl Jul 7, 2023
fc70523
fix: retry 5 times + wait 45 sec for DAAC
wphyojpl Jul 7, 2023
c39fde2
feat: make re-try configurable
wphyojpl Jul 7, 2023
db64367
feat: make re-try configurable
wphyojpl Jul 7, 2023
6e03fe4
fix: download error (#182)
wphyojpl Jul 10, 2023
4845f19
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jul 10, 2023
b20d4a0
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jul 10, 2023
f668e44
chore: merge from develop
wphyojpl Jul 10, 2023
d1a9b2e
Updated CHANGELOG.md to include Unity Release 23.2 release notes.
ngachung Jul 20, 2023
bc8a922
feat: adding fastapi (in-progress)
wphyojpl Jul 20, 2023
924aa42
feat: adding fastapi (in-progress)
wphyojpl Jul 20, 2023
bf37b6a
fix: get collection creation working (in-progress)
wphyojpl Jul 21, 2023
a2a8718
fix: get collection creation working (in-progress)
wphyojpl Jul 21, 2023
d1a8beb
fix: Update stage out task to read STAC items from STAC catalog (#188)
wphyojpl Jul 21, 2023
46a2f7b
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jul 24, 2023
f803439
fix: add root link in collection stac
wphyojpl Jul 24, 2023
308d840
feat: updating terraform
wphyojpl Jul 24, 2023
74af0be
fix: actual creation has some error
wphyojpl Jul 25, 2023
774f3a8
fix: json to invoke magnum lambda is updated
wphyojpl Jul 25, 2023
b4d6948
chore: update log statement
wphyojpl Jul 26, 2023
3492b99
fix: attempting bugfix
wphyojpl Jul 26, 2023
2260947
fix: some weird error. updating function name to confirm it
wphyojpl Jul 26, 2023
a3217d4
fix: teraform error due to partial update only
wphyojpl Jul 26, 2023
3a297d8
fix: dummy commit
wphyojpl Jul 26, 2023
16bb957
fix: allow slash and no slash
wphyojpl Jul 26, 2023
72febf0
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
64fb114
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
c9454ad
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
e57d88c
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
b27be2a
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
bb029c6
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
30d7133
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
cb13811
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
64885de
fix: trying slash and no-slash
wphyojpl Jul 31, 2023
68f5ff3
feat: add uds endpoint tests
wphyojpl Jul 31, 2023
33a3089
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Jul 31, 2023
f3e4f26
chore: merge from fastapi
wphyojpl Jul 31, 2023
3fa9081
feat: am-uds-dapa comes from config with defaulted to that string
wphyojpl Jul 31, 2023
1beaabb
Merge branch 'fastapi' of github.com:unity-sds/unity-data-services in…
wphyojpl Jul 31, 2023
9f762a2
feat: adding fastapi
wphyojpl Jul 31, 2023
38e265b
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Aug 7, 2023
2f88121
feat: endpoint to setup es
wphyojpl Aug 7, 2023
8724fad
feat: Using Fastapi for all API endpoints (#190)
wphyojpl Aug 7, 2023
822d7d9
fix: merge from develop
wphyojpl Aug 7, 2023
1839e1a
fix: make venjue specific ssm params
wphyojpl Aug 7, 2023
6b3e249
fix: need to dump model to become json
wphyojpl Aug 7, 2023
3359c6e
fix: json validator error
wphyojpl Aug 8, 2023
b106116
fix: json validator error
wphyojpl Aug 8, 2023
3d8243e
feat: adding auth to endpoints
wphyojpl Aug 8, 2023
4e43eb6
fix: typo
wphyojpl Aug 8, 2023
8281c4d
fix: add some logs
wphyojpl Aug 8, 2023
14f6815
fix: add some logs
wphyojpl Aug 8, 2023
655819e
fix: add some logs
wphyojpl Aug 8, 2023
b6f7d0d
fix: update new collection to custom collection table
wphyojpl Aug 8, 2023
8919baa
fix: need token now for internal call
wphyojpl Aug 8, 2023
b1c776e
fix: need token now for internal call
wphyojpl Aug 8, 2023
def53b8
fix: need token now for internal call
wphyojpl Aug 8, 2023
3813adc
fix: need token now for internal call
wphyojpl Aug 8, 2023
694699e
fix: need token now for internal call
wphyojpl Aug 8, 2023
33204cf
fix: no need to ignore case
wphyojpl Aug 8, 2023
8166883
chore: merge from develop
wphyojpl Aug 9, 2023
05c460c
fix: S3 upload structure (add collection folder) (#193)
wphyojpl Aug 16, 2023
2c3aa62
fix: Cataloging large number asynchronously by batch + download is st…
wphyojpl Aug 16, 2023
861be39
chore: merged from develop
wphyojpl Aug 16, 2023
a702b37
Merge branch 'develop' of github.com:unity-sds/unity-data-services in…
wphyojpl Aug 16, 2023
25b3c51
Merge branch 'develop' into elasticsearch.lib
wphyojpl Sep 13, 2023
cdda67a
chore: merge from develop
wphyojpl Sep 13, 2023
ea3faf3
Merge branch 'elasticsearch.lib' of github.com:unity-sds/unity-data-s…
wphyojpl Sep 13, 2023
3e0d0a7
chore: pull from develop
wphyojpl Sep 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [5.4.0] - 2023-08-16
### Added
- [#95](https://github.com/unity-sds/unity-data-services/pull/95) feat: elasticsearch branch

## [5.3.3] - 2023-09-18
### Changed
- [#204](https://github.com/unity-sds/unity-data-services/pull/204) chore: clean.up - remove old codes
Expand Down
13 changes: 13 additions & 0 deletions ci.cd/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,21 @@ update_lambda_function_mcp_dev_6:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-metadata_s4pa_generate_cmr --publish &>/dev/null
update_lambda_function_mcp_dev_7:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-metadata_stac_generate_cmr --publish &>/dev/null
update_lambda_function_mcp_dev_auth_list:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_auth_list --publish &>/dev/null
update_lambda_function_mcp_dev_auth_add:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_auth_add --publish &>/dev/null
update_lambda_function_mcp_dev_auth_delete:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_auth_delete --publish &>/dev/null

update_lambda_function_mcp_dev_uds_api:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-uds_api_1 --publish &>/dev/null

update_lambda_function_mcp_sbx_uds_api:
aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-sbx-cumulus-uds_api_1 --publish &>/dev/null

mcp_dev: upload_lambda_mcp_dev update_lambda_function_mcp_dev_auth_list update_lambda_function_mcp_dev_auth_add update_lambda_function_mcp_dev_auth_delete

mcp_dev_fastapi: upload_lambda_mcp_dev update_lambda_function_mcp_dev_uds_api

mcp_sbx_fastapi: upload_lambda_mcp_dev update_lambda_function_mcp_sbx_uds_api
Empty file.
181 changes: 181 additions & 0 deletions cumulus_lambda_functions/cumulus_auth_crud/auth_crud.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
import json
import os

from cumulus_lambda_functions.lib.authorization.uds_authorizer_abstract import UDSAuthorizorAbstract
from cumulus_lambda_functions.lib.authorization.uds_authorizer_factory import UDSAuthorizerFactory
from cumulus_lambda_functions.lib.json_validator import JsonValidator

from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator
from cumulus_lambda_functions.lib.uds_db.db_constants import DBConstants
from cumulus_lambda_functions.lib.utils.lambda_api_gateway_utils import LambdaApiGatewayUtils

LOGGER = LambdaLoggerGenerator.get_logger(__name__, LambdaLoggerGenerator.get_level_from_env())


delete_schema = {
'type': 'object',
'required': ['tenant', 'venue', 'group_name'],
'properties': {
'tenant': {'type': 'string'},
'venue': {'type': 'string'},
'group_name': {'type': 'string'},
}
}

list_schema = {
'type': 'object',
'properties': {
'tenant': {'type': 'string'},
'venue': {'type': 'string'},
'group_names': {
'type': 'array',
'items': {'type': 'string'},
'minItems': 1,
},
}
}

add_schema = {
'type': 'object',
'required': ['tenant', 'venue', 'group_name', 'actions', 'resources'],
'properties': {
'tenant': {'type': 'string'},
'venue': {'type': 'string'},
'group_name': {'type': 'string'},
'actions': {
'type': 'array',
'items': {
'type': 'string',
'enum': [DBConstants.create, DBConstants.delete, DBConstants.update, DBConstants.read]
},
'minItems': 1,
},
'resources': {
'type': 'array',
'items': {'type': 'string'},
'minItems': 1,
},
}

}


class AuthCrud:
def __init__(self, event):
required_env = ['ES_URL', 'ADMIN_COMMA_SEP_GROUPS']
if not all([k in os.environ for k in required_env]):
raise EnvironmentError(f'one or more missing env: {required_env}')
self.__admin_groups = [k.strip() for k in os.getenv('ADMIN_COMMA_SEP_GROUPS').split(',')]
self.__event = event
self.__request_body = {}
self.__es_url = os.getenv('ES_URL')
self.__es_port = int(os.getenv('ES_PORT', '443'))
self.__authorizer: UDSAuthorizorAbstract = UDSAuthorizerFactory() \
.get_instance(UDSAuthorizerFactory.cognito,
es_url=self.__es_url,
es_port=self.__es_port
)
self.__lambda_utils = LambdaApiGatewayUtils(self.__event, 10)

def __is_admin(self):
auth_info = self.__lambda_utils.get_authorization_info()
belonged_admin_groups = list(set(self.__admin_groups) & set(auth_info['ldap_groups']))
return len(belonged_admin_groups) > 0

def __load_request_body(self):
if 'body' in self.__event and self.__event['body'] is not None:
self.__request_body = json.loads(self.__event['body'])
return

def list_all_record(self):
if not self.__is_admin():
return {
'statusCode': 403,
'body': f'user is not in admin groups: {self.__admin_groups}'
}
self.__load_request_body()
all_records = self.__authorizer.list_groups(
tenant=self.__request_body['tenant'] if 'tenant' in self.__request_body else None,
venue=self.__request_body['venue'] if 'venue' in self.__request_body else None,
ldap_group_names=self.__request_body['group_names'] if 'group_names' in self.__request_body else None,
)
return {
'statusCode': 200,
'body': json.dumps(all_records)
}

def add_new_record(self):
if not self.__is_admin():
return {
'statusCode': 403,
'body': f'user is not in admin groups: {self.__admin_groups}'
}
self.__load_request_body()
body_validator_result = JsonValidator(add_schema).validate(self.__request_body)
if body_validator_result is not None:
LOGGER.error(f'invalid add body: {body_validator_result}. request_body: {self.__request_body}')
return {
'statusCode': 500,
'body': f'invalid add body: {body_validator_result}. request_body: {self.__request_body}'
}
self.__authorizer.add_authorized_group(
action=self.__request_body['actions'],
resource=self.__request_body['resources'],
tenant=self.__request_body['tenant'],
venue=self.__request_body['venue'],
ldap_group_name=self.__request_body['group_name'],
)
return {
'statusCode': 200,
'body': 'inserted'
}

def update_record(self):
if not self.__is_admin():
return {
'statusCode': 403,
'body': f'user is not in admin groups: {self.__admin_groups}'
}
self.__load_request_body()
body_validator_result = JsonValidator(add_schema).validate(self.__request_body)
if body_validator_result is not None:
LOGGER.error(f'invalid update body: {body_validator_result}. request_body: {self.__request_body}')
return {
'statusCode': 500,
'body': f'invalid update body: {body_validator_result}. request_body: {self.__request_body}'
}
self.__authorizer.update_authorized_group(
action=self.__request_body['actions'],
resource=self.__request_body['resources'],
tenant=self.__request_body['tenant'],
venue=self.__request_body['venue'],
ldap_group_name=self.__request_body['group_name'],
)
return {
'statusCode': 200,
'body': 'updated'
}

def delete_record(self):
if not self.__is_admin():
return {
'statusCode': 403,
'body': f'user is not in admin groups: {self.__admin_groups}'
}
self.__load_request_body()
body_validator_result = JsonValidator(delete_schema).validate(self.__request_body)
if body_validator_result is not None:
LOGGER.error(f'invalid delete body: {body_validator_result}. request_body: {self.__request_body}')
return {
'statusCode': 500,
'body': f'invalid delete body: {body_validator_result}. request_body: {self.__request_body}'
}
self.__authorizer.delete_authorized_group(
tenant=self.__request_body.get('tenant'),
venue=self.__request_body.get('venue'),
ldap_group_name=self.__request_body.get('group_name'),
)
return {
'statusCode': 200,
'body': 'deleted'
}
22 changes: 22 additions & 0 deletions cumulus_lambda_functions/cumulus_auth_crud/lambda_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from cumulus_lambda_functions.cumulus_auth_crud.auth_crud import AuthCrud
from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator


def auth_list(event, context):
LambdaLoggerGenerator.remove_default_handlers()
return AuthCrud(event).list_all_record()


def auth_add(event, context):
LambdaLoggerGenerator.remove_default_handlers()
return AuthCrud(event).add_new_record()


def auth_update(event, context):
LambdaLoggerGenerator.remove_default_handlers()
return AuthCrud(event).update_record()


def auth_delete(event, context):
LambdaLoggerGenerator.remove_default_handlers()
return AuthCrud(event).delete_record()
Empty file.
38 changes: 38 additions & 0 deletions cumulus_lambda_functions/cumulus_es_setup/es_mappings.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
alias_pointer = {
"actions" : [
{"add" : {"index" : "authorization_mappings_v1", "alias" : "authorization_mappings"}},
{"add" : {"index" : "unity_collections_v1", "alias" : "unity_collections"}}
]
}

authorization_mappings_v1 = {
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
},
"mappings": {
"properties": {
"action": {"type": "keyword"},
"collection_map": {"type": "keyword"},
"user_group": {"type": "keyword"},
"tenant": {"type": "keyword"},
"tenant_venue": {"type": "keyword"}
}
}
}

unity_collections_v1 = {
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
},
"mappings": {
"properties": {
"collection_id": {"type": "keyword"},
"bbox": {"type": "geo_shape"},
"granule_count": {"type": "integer"},
"start_time": {"type": "long"},
"end_time": {"type": "long"}
}
}
}
43 changes: 43 additions & 0 deletions cumulus_lambda_functions/cumulus_es_setup/es_setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import os

from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator

from cumulus_lambda_functions.lib.aws.es_abstract import ESAbstract

from cumulus_lambda_functions.lib.aws.es_factory import ESFactory

from cumulus_lambda_functions.lib.uds_db.db_constants import DBConstants
from cumulus_lambda_functions.cumulus_es_setup import es_mappings
LOGGER = LambdaLoggerGenerator.get_logger(__name__, LambdaLoggerGenerator.get_level_from_env())


class SetupESIndexAlias:
def __init__(self):
required_env = ['ES_URL']
if not all([k in os.environ for k in required_env]):
raise EnvironmentError(f'one or more missing env: {required_env}')
self.__es: ESAbstract = ESFactory().get_instance('AWS',
index=DBConstants.collections_index,
base_url=os.getenv('ES_URL'),
port=int(os.getenv('ES_PORT', '443'))
)

def get_index_mapping(self, index_name: str):
if not hasattr(es_mappings, index_name):
raise ValueError(f'missing index_name: {index_name}')
index_json = getattr(es_mappings, index_name)
return index_json

def start(self):
if not hasattr(es_mappings, 'alias_pointer'):
raise ValueError(f'missing alias_pointer')
alias_json = getattr(es_mappings, 'alias_pointer')
alias_json = [k['add'] for k in alias_json['actions']]
for each_action in alias_json:
current_index = each_action['index']
current_alias = each_action['alias']
LOGGER.debug(f'working on {current_index}')
index_json = self.get_index_mapping(current_index)
self.__es.create_index(current_index, index_json)
self.__es.create_alias(current_index, current_alias)
return self
13 changes: 13 additions & 0 deletions cumulus_lambda_functions/cumulus_es_setup/lambda_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from cumulus_lambda_functions.cumulus_es_setup.es_setup import SetupESIndexAlias
from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator


def lambda_handler(event, context):
"""
{'cma': {'task_config': {'bucket': '{$.meta.buckets.internal.name}', 'collection': '{$.meta.collection}', 'cumulus_message': {'outputs': [{'source': '{$.files}', 'destination': '{$.payload}'}]}}, 'event': {'cumulus_meta': {'cumulus_version': '10.0.1', 'execution_name': 'c6d885dc-b4b2-4eb0-b22e-b6f58a7a0870', 'message_source': 'sfn', 'queueExecutionLimits': {'https://sqs.us-west-2.amazonaws.com/884500545225/am-uds-dev-cumulus-backgroundProcessing': 5}, 'state_machine': 'arn:aws:states:us-west-2:884500545225:stateMachine:am-uds-dev-cumulus-IngestGranule', 'system_bucket': 'am-uds-dev-cumulus-internal', 'workflow_start_time': 1646785175509, 'parentExecutionArn': 'arn:aws:states:us-west-2:884500545225:execution:am-uds-dev-cumulus-DiscoverGranules:885483b4-ba55-4db1-b197-661e1e595a45', 'queueUrl': 'arn:aws:sqs:us-west-2:884500545225:am-uds-dev-cumulus-startSF'}, 'exception': 'None', 'meta': {'buckets': {'internal': {'name': 'am-uds-dev-cumulus-internal', 'type': 'internal'}, 'protected': {'name': 'am-uds-dev-cumulus-protected', 'type': 'protected'}}, 'cmr': {'clientId': 'CHANGEME', 'cmrEnvironment': 'UAT', 'cmrLimit': 100, 'cmrPageSize': 50, 'oauthProvider': 'earthdata', 'passwordSecretName': 'am-uds-dev-cumulus-message-template-cmr-password20220216072916956000000002', 'provider': 'CHANGEME', 'username': 'username'}, 'collection': {'name': 'ATMS_SCIENCE_Group_2011', 'version': '001', 'process': 'modis', 'granuleId': '^P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}0$', 'granuleIdExtraction': '(P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}0).+', 'sampleFileName': 'P1570515ATMSSCIENCEAXT11344000000001.PDS', 'duplicateHandling': 'replace', 'url_path': '{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}', 'provider_path': '/data/SNPP_ATMS_Level0_T/ATMS_SCIENCE_Group/2011/', 'files': [{'bucket': 'internal', 'regex': '^P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}00\\.PDS$', 'sampleFileName': 'P1570515ATMSSCIENCEAXT11344000000000.PDS', 'type': 'data'}, {'bucket': 'internal', 'regex': '^P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}01\\.PDS$', 'sampleFileName': 'P1570515ATMSSCIENCEAXT11344000000001.PDS', 'type': 'metadata'}, {'bucket': 'internal', 'regex': '^P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}01\\.PDS\\.xml$', 'sampleFileName': 'P1570515ATMSSCIENCEAXT11344000000001.PDS.xml', 'type': 'metadata'}, {'bucket': 'internal', 'regex': '^P[0-9]{3}[0-9]{4}[A-Z]{13}T[0-9]{12}0\\.cmr\\.xml$', 'sampleFileName': 'P1570515ATMSSCIENCEAXT11344000000001.PDS.xml', 'type': 'metadata'}], 'updatedAt': 1646326197526, 'createdAt': 1646258167624}, 'distribution_endpoint': 's3://am-uds-dev-cumulus-internal/', 'launchpad': {'api': 'launchpadApi', 'certificate': 'launchpad.pfx', 'passphraseSecretName': ''}, 'provider': {'password': 'AQICAHhSagsGDAl5tQWM010IEvxKgj2LcsNub5v5FHoRpOjXcQHFbE4iMnF/W0Y/NrsYvrfHAAAAajBoBgkqhkiG9w0BBwagWzBZAgEAMFQGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMLaH13SdxPXREjXLtAgEQgCfA+lEu2c/xLTGwJsbtKlXJbKDy4pwV+rS3BnJqgBoLLMQZqOdoFhk=', 'host': 'snppl0.gesdisc.eosdis.nasa.gov', 'updatedAt': 1646244053419, 'protocol': 'https', 'createdAt': 1646244053419, 'encrypted': True, 'username': 'AQICAHhSagsGDAl5tQWM010IEvxKgj2LcsNub5v5FHoRpOjXcQGRoY5EBMpvvyMASUowBM61AAAAYzBhBgkqhkiG9w0BBwagVDBSAgEAME0GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9OhRHwTuxiz74q4UAgEQgCDEHOhsVG6+LqXfnlw+Z3Wg9MDOCd9/K5/X5j3tPJYkaA==', 'allowedRedirects': ['https://urs.earthdata.nasa.gov', 'urs.earthdata.nasa.gov'], 'id': 'snpp_provider_02', 'globalConnectionLimit': 10}, 'stack': 'am-uds-dev-cumulus', 'template': 's3://am-uds-dev-cumulus-internal/am-uds-dev-cumulus/workflow_template.json', 'workflow_name': 'IngestGranule', 'workflow_tasks': {'SyncGranule': {'name': 'am-uds-dev-cumulus-SyncGranule', 'version': '$LATEST', 'arn': 'arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-SyncGranule'}}, 'staticValue': 'aStaticValue', 'interpolatedValueStackName': 'am-uds-dev-cumulus', 'input_granules': [{'granuleId': 'P1570515ATMSSCIENCEAXT1134912000000', 'dataType': 'ATMS_SCIENCE_Group_2011', 'version': '001', 'files': [{'bucket': 'am-uds-dev-cumulus-internal', 'key': 'file-staging/am-uds-dev-cumulus/ATMS_SCIENCE_Group_2011___001/P1570515ATMSSCIENCEAXT11349120000000.PDS', 'source': 'data/SNPP_ATMS_Level0_T/ATMS_SCIENCE_Group/2011/349//P1570515ATMSSCIENCEAXT11349120000000.PDS', 'fileName': 'P1570515ATMSSCIENCEAXT11349120000000.PDS', 'type': 'data', 'size': 744}, {'bucket': 'am-uds-dev-cumulus-internal', 'key': 'file-staging/am-uds-dev-cumulus/ATMS_SCIENCE_Group_2011___001/P1570515ATMSSCIENCEAXT11349120000001.PDS', 'source': 'data/SNPP_ATMS_Level0_T/ATMS_SCIENCE_Group/2011/349//P1570515ATMSSCIENCEAXT11349120000001.PDS', 'fileName': 'P1570515ATMSSCIENCEAXT11349120000001.PDS', 'type': 'metadata', 'size': 18084600}, {'bucket': 'am-uds-dev-cumulus-internal', 'key': 'file-staging/am-uds-dev-cumulus/ATMS_SCIENCE_Group_2011___001/P1570515ATMSSCIENCEAXT11349120000001.PDS.xml', 'source': 'data/SNPP_ATMS_Level0_T/ATMS_SCIENCE_Group/2011/349//P1570515ATMSSCIENCEAXT11349120000001.PDS.xml', 'fileName': 'P1570515ATMSSCIENCEAXT11349120000001.PDS.xml', 'type': 'metadata', 'size': 9526}], 'sync_granule_duration': 9822, 'createdAt': 1647386972717}], 'process': 'modis'}, 'payload': {}, 'replace': {'Bucket': 'am-uds-dev-cumulus-internal', 'Key': 'events/5d8edf37-0a18-4af5-a76f-7c2091cdd1e2', 'TargetPath': '$.payload'}}}}
:param event:
:param context:
:return:
"""
LambdaLoggerGenerator.remove_default_handlers()
return SetupESIndexAlias().start()
Loading