chore: clean.up - remove old codes (#204)

* testing github actions

* fix: tweak for github action

* fix: still testing github actions

* fix: remove name from steps

* fix: tweaking python3 command

* fix: remove pip to see if requirements.txt error goes away

* feat: add release + upload procedure

* fix: remove double quotes

* fix: fixing `" error

* fix: still fixing getting software_version line

* fix: still fixing software_version

* fix: breaking the script to debug

* fix: more debugging stmt

* fix: still tweaking around

* fix: more tweaking

* fix: still tweaking

* feat: add terraform module for lambdas

* feat: update ci/cd to include terraform

* chore: bump version

* fix: add terraform zip as asset

* fix: upload multiple assets

* fix: still trying to upload multiple assets

* feat: bypass A&A in cumulus

* fix: update hardcoded prefix to prefix come from config + add missing ENV in terraform

* feat: add collection_id to the query

* chore: existing api gateway yaml for dapa

* feat: add collectionId as path parameter

* feat: add new lambda for collections

* fix: update terraform module

* fix: collection lambda error

* chore: bump version

* fix: update lambda executing role

* fix: collection listing lambda bug

* feat: add code for granules download + docker for it

* feat: update swagger doc for api gateway

* fix: zip terraform in current directory. not in a folder

* fix: add missing terraform output

* feat: add bearer token to docker call

* chore: readme on how to get cognito token

* feat: add l1a metadata extraction code

* feat: add level1 generate cmr to terraform

* chore: update version

* fix: terraform invalid character error

* fix: terraform unsupported argument

* fix: terraform unsupported argument

* fix: use granuleId value from cma

* chore: github workflow to trigger only on main branch

* feat: add docker build github action

* Create dockerbuild.yml

* fix: hardcoding username for now

* fix: testing with dockerhub + fixed name

* fix: trying docker build

* fix: still figuring out docker

* fix: add missing Dockerfile

* fix: using absolute path

* fix: updating context

* feat: use version from setup.py

* feat: add file configuration as links in collection to STAC

* feat: add granuleId and others to summaries

* feat: allow bucket+key / url_path. add bucket+key if url_path is present

* fix: add additional metadata in cmr xml output to match other file to avoid NULL error in workflow

* chore: bump version

* feat: dapa endpoint to ingest to cnm

* feat: add sns msg

* fix: use main branch to deploy docker (which is not working)

* Proposed CONTRIBUTING.md file / template

* fix: get collection id only. not versin + add sample test input json

* fix: github workflow needs to run only on main branch

* feat: docker script to upload granules via cnm + cnm ingest lambda bugfix

* feat: update terraform to include a new lambda

* breaking: update docker entry script

* fix: missing path in upload script

* fix: checking if querystring is NULL

* fix: added lambda_processing_role_arn as a variable to resolve terraform dependency issues

* fix: version bump

* release/1.5.14 (#56)

* feat: docker script to upload granules via cnm + cnm ingest lambda bugfix

* feat: update terraform to include a new lambda

* breaking: update docker entry script

* fix: missing path in upload script

* fix: checking if querystring is NULL

* fix: added lambda_processing_role_arn as a variable to resolve terraform dependency issues

* fix: version bump

Co-authored-by: Wai Phyo <wai.phyo@jpl.nasa.gov>

* change: get github-actions to publish docker image to ghcr.io (#57)

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker github action

* fix: docker build github action

* fix: docker build github action

* fix: docker build github action

* fix: docker build github action

* fix: docker build github action

* chore: run github action in main branch

* ADD: adding extra keys based on extra-config in input event (#59)

* feat: adding extra keys based on extra-config in input event

* chore: reformat code

* fix: dataVersion = collection_version for cnm SNS msg

* fix: validate if datetime is null

* ADD: user username & password to get token from cognito (#61)

* feat: user username & password to get token from cognito

* chore: update changelog to match with version

* chore: change release order

* fix: l1A granule id is `<collection-id>___<collection-version>:<granule-id>` not to duplicate re-runs

* chore: update error description

* feat: add size, prev, next url for granules

* feat: add stats for collection + size. fix: check if queryString is NULL

* fix: encap code in try-catch to avoid `internal server error`

* Updated README.md template (#69)

* fix: add err msg when next & prev url does not generate + update makefile for local

* feat: STAC style pagination (#70)

* fix: add err msg when next & prev url does not generate + update makefile for local

* feat: update to STAC specific schema ref from CMR

* feat: make pagination class

* breaking: pystac validation test + upload / download docker unit test + update collection STAC schema update (#71)

* breaking: testing with pystac + make changes accordingly

* chore: convert manual test to unittest case

* fix: update test case

* fix: update upload granule to match new pystac collection schema

* chore: stashing collection transformer to use pystac (in-progress)

* chore: remove unused file

* chore: update gitignore file

* fix: update test cases

* chore: Open api yaml for documentation (#72)

* chore: add open_api doc

* fix: update open api yaml errors

* chore: add test yaml

* chore: add test yaml

* chore: add test yaml

* chore: add test yaml

* feat: add other venue urls

* chore: add unit test for param store cognito login (#74)

* chore: move the unit test (#75)

* Update setup.py

* fix: GitHub.action.update (#76)

* fix: disable building upon pull request

* chore: bump version

* /version 1.7.0 release

* fix: level.1.a.missing.filename (#80)

* fix: loop all files before returning metadata file

* chore: bump version

* fix: not honoring offset and limit in Collection query (#82)

* fix: not assigning limit and offset from api endpoint

* chore: bump version

* feat: Collection Creation endpoint with DAPA format (#79)

* chore: move the unit test

* feat: add create collection for cumulus

* feat: adding stac to cumulus for json

* feat: finished transforming from stac to cumulus

* fix: try creating collection with min requirement

* feat: add class to create dapa collection obj

* fix: add class to create dapa collection object for unity

* feat: add class to create STAC collection

* feat: update to python3.9 + new lambda for collection creation

* chore: python3.9 for github action

* fix: ci/cd on develop branch with additional release postfix

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* fix: testing different output number

* chore: add testcase.. in progress

* fix: wrong handler for new lambda

* fix: require libraries at the worng location

* fix: add mock url if url_path is misisng

* feat: add rule when creating a collection

* fix: wrong quotes in terraform

* fix: use thread to run the actual collection creation in background thread

* fix: revert background process. does not work as expected in lambda

* fix: addig facade for collection creation

* fix: api gateway wants string for body. not dict

* fix: no need to hash the dict

* feat: add single collection details

* feat: add single collection detail logic

* fix: update testcase + yaml for api gateway

* feat: delete collection if rule creation fails

* chore: update version

* fix: check pathParameters is None (#89)

* chore: bump version + update changelog (#90)

* fix: Completed.granules.only + validate start + end time for STAC (#93)

* feat: add completed status on granules

* fix: check start & end datetime before pushing add them to stac + defaulted it to unix 0

* Added DAPA lambdas invoke arn values to parameter store

* Updated to store function name instead.

* Updated to store function name instead.

* fix: accept provider from ENV or optionally from user call

* chore: update test case

* fix: add provider in rules

* chore: update changelog

* Updated CHANGELOG.md

* fix: status=completed is only for granules, not for collections

* feat: return upload result (#103)

* feat: return upload result

* chore: update version

* feat: added Updated time in collection & item STAC dictionaries (#104)

* feat: add updated key in granules and collections

* chore: use pystac collection object to create a collection

* chore: use pystac to transform to item

* chore: update changelog

* chore: update version

* fix: update dockerfile base image to 3.9 (#112)

* fix: update dockerfile base image to 3.9

* fix: local manual test with latest image

* fix: new stage-in stage-out

* fix: remove old download scripts

* fix: docker scripts for split download + search

* chore: update version + change log

* feat: add upload step + test

* feat: add catalog stage

* chore: remove old class

* chore: update sample docker-compose files

* fix: print method output in docker + tweak docker compose files

* chore: update changelog

* feat: add CMR granules search which returns STAC

* breaking: configurable file postfixes for PDS metadata extraction (#124)

* feat: configurable file postfixes for PDS metadata extraction

* fix: some bugfix + update manual deployment update

* breaking: rename lvl0 and lvl1 to s4pa and cas

* fix: rename output variables

* chore: update contributing

* feat: reduce pystac length by keeping only data asset (#126)

* feat: reduce pystac length by keeping only data asset

* chore: update version

* feat: granules query pagination (#131)

* feat: reduce pystac length by keeping only data asset

* chore: update version

* feat: pagination for cumulus granules search

* feat: add pagination for CMR + bugfix in Cumulus pagination

* chore: update version

* fix: add filter stac items to assets only

* fix: add missing ENV in sample yaml

* feat: add EDL token class

* feat: adding DAAC download in docker stage in (in-progress)

* feat: add option to parse downloading stac from file (#134)

* feat: add option to parse downloading stac from file

* fix: version increment error + update change log

* fix: uncomment temporal in CMR granules search (#136)

* fix: uncomment temporal

* chore: update changelog

* fix: some typo in download daac

* chore: add some TODOs

* fix: update client id

* fix: update client id

* chore: update changelog

* feat: Checkout stage with STAC catalog json (#138)

* feat: reduce pystac length by keeping only data asset

* chore: update version

* feat: pagination for cumulus granules search

* feat: add pagination for CMR + bugfix in Cumulus pagination

* chore: update version

* fix: add filter stac items to assets only

* feat: upload by stac catalog

* chore: update version

* feat: daac download from stac json file

* chore: update version

* fix: downloaded stac to return local absolute path

* chore: update PR link

* feat: add metadata extraction from stac + refactor

* chore: refactor

* chore: refactor

* fix: error during refactor

* chore: still refactoring

* fix: update code for stac lambda

* feat: add terraform

* chore: update version

* feat: optional output to file

* fix: add unit test + create folder if needed

* fix: tweak it a bit to avoid writing it twice in case some steps are calling one another

* chore: update readme

* fix: use cas structure to generate metadata for stac (#148)

* fix: use cas to generate metadata for stac

* chore: update Makefile

* chore: update version

* fix: need to run for every file so that every file is updated even after correct file is found

* fix: allow catalog stage input from file

* chore: update changelog

* fix: remove local var. using parent var

* fix: production datetime not in +00:00 format

* chore: update version + readme

* feat: add public http download class

* breaking: search return entire feature-collection

* breaking: daac downloader accepts only features collection

* chore: update daac testcase

* feat: refactor download classes to use inheritance

* fix: http downloader had a typo bug

* chore: update readme

* chore: update readme

* feat: new upload handler for relative paths + user granule-id from the granule-stac-json

* feat: output to be an item-collection

* feat: update catalog to accept item collection

* chore: update version

* breaking: removing old upload styles + default it to new style if missing

* chore: delete old classes

* chore: update version since it is breaking change now

* fix: convert all outputs into json str

* chore: update version

* chore: update PR number

* feat: added filter keyword in granules endpoint + repeatedly checking with time boundary for cataloging result (#156)

* feat: first cut of checking cataloging granules

* fix: add missing file

* fix: unit test + bugfix

* feat: update response json in catalog

* chore: update version

* chore: update documentation

* chore: remove invalid MD in changelog

* fix: update due to feature-collection update from other PR

* feat: parallelize download (#169)

* feat: starting parallel job manager

* fix: get parallel process working for download

* feat: update testcases

* chore: update version

* chore: update readme

* feat: add configurable parallel count

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* fix: download error (#182)

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* fix: retry on Temporary failure in name resolution

* chore: update version

* fix: add new test case for error

* Updated CHANGELOG.md to include Unity Release 23.2 release notes.

* feat: adding fastapi (in-progress)

* fix: get collection creation working (in-progress)

* fix: Update stage out task to read STAC items from STAC catalog (#188)

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* fix: user item, not child in rel

* chore: update PR link

* feat: Using Fastapi for all API endpoints (#190)

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* feat: adding fastapi (in-progress)

* fix: get collection creation working (in-progress)

* fix: add root link in collection stac

* feat: updating terraform

* fix: actual creation has some error

* fix: json to invoke magnum lambda is updated

* chore: update log statement

* fix: attempting bugfix

* fix: some weird error. updating function name to confirm it

* fix: teraform error due to partial update only

* fix: dummy commit

* fix: allow slash and no slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* fix: trying slash and no-slash

* feat: add uds endpoint tests

* feat: am-uds-dapa comes from config with defaulted to that string

* fix: S3 upload structure (add collection folder) (#193)

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* feat: adding fastapi (in-progress)

* fix: get collection creation working (in-progress)

* feat: add collection folder in s3 upload

* fix: update note

* fix: update wrong PR number

* fix: update missed file during merge

* fix: remove old test case

* fix: Cataloging large number asynchronously by batch + download is stuck when there are large number of files (#194)

* fix: retry 5 times + wait 45 sec for DAAC

* feat: make re-try configurable

* feat: adding fastapi (in-progress)

* fix: get collection creation working (in-progress)

* feat: default poll checking to 0 times + conflict bug in dapa client

* feat: add facade endpoint + real endpoint for cnm

* fix: update version

* fix: need to call PUT. not POST

* fix: update test case

* chore: add large data catalog testcase

* fix: use Manager().Q instead of normal Q to fix it for download stuck for large granules

* chore: adding support files for test

* fix: calling catalog api in batch

* chore: update changelog

* chore: add missing test file

* fix: mismatch version

* chore: remove old code

* chore: move dapa client to stage in-out coz it is only used in that module

* chore: update version

---------

Co-authored-by: Nga Chung <17833879+ngachung@users.noreply.github.com>
Co-authored-by: nchung <nga.t.chung@jpl.nasa.gov>
Co-authored-by: Rishi Verma <riverma@apache.org>

Author: 3 people

CHANGELOG.md

@@ -5,10 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [5.3.3] - 2023-09-18
+### Changed
+- [#204](https://github.com/unity-sds/unity-data-services/pull/204) chore: clean.up - remove old codes
+
 ## [5.3.2] - 2023-08-28
 ### Changed
 - [#200](https://github.com/unity-sds/unity-data-services/pull/200) fix: parallelize upload
-- 
+
 ## [5.3.1] - 2023-08-16
 ### Changed
 - [#194](https://github.com/unity-sds/unity-data-services/pull/194) fix: Cataloging large number asynchronously by batch + download is stuck when there are large number of files

ci.cd/Makefile

@@ -4,7 +4,7 @@ export VERSION ?= latest
 
 
 all: build_lambda upload_lambda update_lambda_function build_docker
-local: build_lambda upload_lambda update_lambda_function_1 update_lambda_function_2 update_lambda_function_3
+local: build_lambda upload_lambda
 build_docker:
 	docker build --no-cache -t "$(IMAGE_PREFIX)/$(NAME):$(VERSION)" -f docker/Dockerfile_download_granules.jpl .
 
@@ -19,28 +19,9 @@ build_lambda_public:
 
 upload_lambda:
 	aws --profile saml-pub s3 cp cumulus_lambda_functions_deployment.zip s3://am-uds-dev-cumulus-tf-state/unity_cumulus_lambda/
-update_lambda_function_1:
-	aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_dapa --publish &>/dev/null
-update_lambda_function_2:
-	aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_granules_dapa --publish &>/dev/null
-update_lambda_function_3:
-	aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_ingest_cnm_dapa --publish &>/dev/null
-update_lambda_function_4:
-	aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_creation_dapa --publish &>/dev/null
-update_lambda_function_5:
-	aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_creation_dapa_facade --publish &>/dev/null
-
 
 upload_lambda_mcp_dev:
 	aws s3 cp cumulus_lambda_functions_deployment.zip s3://uds-dev-cumulus-public/unity_cumulus_lambda/
-update_lambda_function_mcp_dev_1:
-	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_collections_dapa --publish &>/dev/null
-update_lambda_function_mcp_dev_2:
-	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_granules_dapa --publish &>/dev/null
-update_lambda_function_mcp_dev_4:
-	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_collections_creation_dapa --publish &>/dev/null
-update_lambda_function_mcp_dev_5:
-	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-cumulus_collections_creation_dapa_facade --publish &>/dev/null
 update_lambda_function_mcp_dev_6:
 	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-metadata_s4pa_generate_cmr --publish &>/dev/null
 update_lambda_function_mcp_dev_7:
@@ -50,6 +31,3 @@ update_lambda_function_mcp_dev_uds_api:
 	aws lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket uds-dev-cumulus-public --function-name arn:aws:lambda:us-west-2:237868187491:function:uds-dev-cumulus-uds_api_1 --publish &>/dev/null
 
 mcp_dev_fastapi: upload_lambda_mcp_dev update_lambda_function_mcp_dev_uds_api
-mcp_dev: upload_lambda_mcp_dev update_lambda_function_mcp_dev_1 update_lambda_function_mcp_dev_2 update_lambda_function_mcp_dev_4 update_lambda_function_mcp_dev_5
-
-