Skip to content

Commit

Permalink
GITBOOK-673: stage-out doc updates for aux files
Browse files Browse the repository at this point in the history
  • Loading branch information
@ngachung @gitbook-bot
ngachung authored and gitbook-bot committed Dec 17, 2024
1 parent 8a82d1c commit 639d210
Showing 1 changed file with 19 additions and 19 deletions.
38 changes: 19 additions & 19 deletions developer-docs/data/docs/users-guide/stage-out.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,22 @@ Data can be staged out / uploaded by using the latest DS image available at [htt

Environment variables used with `UPLOAD` command are described below.

| Name | Description | Default Value | Required |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------------- |
| AWS\_ACCESS\_KEY\_ID | AWS access key | | No |
| AWS\_SECRET\_ACCESS\_KEY | AWS secret key associated with the access key | | No |
| AWS\_SESSION\_TOKEN | AWS session token value required when using temporary security credentials | | No |
| AWS\_REGION | AWS region | us-west-2 | No |
| DRY\_RUN | If TRUE, will validate STAC catalog but will not perform actual upload | FALSE | No |
| COLLECTION\_ID | Collection identifier in the format of `<COLLECTION_NAME>___<COLLECTION_VERSION>` | | Yes, only if `GRANULES_UPLOAD_TYPE` = UPLOAD\_AUXILIARY\_FILE\_AS\_GRANULE |
| PROJECT | Name of the mission/project. If collection field in Granule STAC purely includes `<COLLECTION_NAME>___<COLLECTION_VERSION>` then `PROJECT` will be used to create MDPS compliant collection ID by populating `URN:NASA:UNITY:<PROJECT>:<VENUE>:<COLLECTION_NAME>___<COLLECTION_VERSION>` | | No |
| VENUE | Venue (e.g. DEV, TEST, PROD) of the mission/project. If collection field in Granule STAC purely includes `<COLLECTION_NAME>___<COLLECTION_VERSION>` then `VENUE` will be used to create MDPS compliant collection ID by populating `URN:NASA:UNITY:<PROJECT>:<VENUE>:<COLLECTION_NAME>___<COLLECTION_VERSION>` | | No |
| STAGING\_BUCKET | S3 bucket to upload files to | | Yes |
| RESULT\_PATH\_PREFIX | Path to upload STAC FeatureCollection of all uploaded files for auto-cataloging. Final location of STAC FeatureCollection will be `s3://<STAGING_BUCKET>/<RESULT_PATH_PREFIX>/successful_features_<timestamp>.json` | stage\_out | No |
| DELETE\_FILES | If TRUE will delete files after successful upload | FALSE | No |
| GRANULES\_UPLOAD\_TYPE | <p>Needs to be one of the following value</p><ul><li><code>UPLOAD_S3_BY_STAC_CATALOG</code> - for uploading files that have a STAC catalog that points to STAC Feature for each file</li><li><code>UPLOAD_AUXILIARY_FILE_AS_GRANULE</code> - for uploading a directory of files without providing STAC Catalog and STAC Features</li></ul> | UPLOAD\_S3\_BY\_STAC\_CATALOG | No |
| CATALOG\_FILE | | | Yes, if `GRANULES_UPLOAD_TYPE` is `UPLOAD_S3_BY_STAC_CATALOG` |
| BASE\_DIRECTORY | Path to directory containing files to upload | | Yes, if `GRANULES_UPLOAD_TYPE` is `UPLOAD_AUXILIARY_FILE_AS_GRANULE` |
| OUTPUT\_DIRECTORY | Path to directory to write successful and failed FeatureCollection | | Yes |
| LOG\_LEVEL | <p><code>10</code> = DEBUG</p><p><code>20</code> = INFO</p><p><code>30</code> = WARNING</p><p><code>40</code> = ERROR</p><p><code>50</code> = CRITICAL</p><p><code>60</code> = AUDIT</p> | 10 | No |
| PARALLEL\_COUNT | Number of threads used for uploading. Defaults to -1 which is equivalent to CPU count | -1 | No |
| Name | Description | Default Value | Required |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------- |
| AWS\_ACCESS\_KEY\_ID | AWS access key | | No |
| AWS\_SECRET\_ACCESS\_KEY | AWS secret key associated with the access key | | No |
| AWS\_SESSION\_TOKEN | AWS session token value required when using temporary security credentials | | No |
| AWS\_REGION | AWS region | us-west-2 | No |
| DRY\_RUN | If TRUE, will validate STAC catalog but will not perform actual upload | FALSE | No |
| COLLECTION\_ID | Collection identifier in the format of `<COLLECTION_NAME>___<COLLECTION_VERSION>` | | Yes, if `GRANULES_UPLOAD_TYPE` is `UPLOAD_AUXILIARY_FILE_AS_GRANULE` |
| PROJECT | Name of the mission/project. If collection field in Granule STAC purely includes `<COLLECTION_NAME>___<COLLECTION_VERSION>` then `PROJECT` will be used to create MDPS compliant collection ID by populating `URN:NASA:UNITY:<PROJECT>:<VENUE>:<COLLECTION_NAME>___<COLLECTION_VERSION>` | | No |
| VENUE | Venue (e.g. DEV, TEST, PROD) of the mission/project. If collection field in Granule STAC purely includes `<COLLECTION_NAME>___<COLLECTION_VERSION>` then `VENUE` will be used to create MDPS compliant collection ID by populating `URN:NASA:UNITY:<PROJECT>:<VENUE>:<COLLECTION_NAME>___<COLLECTION_VERSION>` | | No |
| STAGING\_BUCKET | S3 bucket to upload files to | | Yes |
| RESULT\_PATH\_PREFIX | Path to upload STAC FeatureCollection of all uploaded files for auto-cataloging. Final location of STAC FeatureCollection will be `s3://<STAGING_BUCKET>/<RESULT_PATH_PREFIX>/successful_features_<timestamp>.json` | stage\_out | No |
| DELETE\_FILES | If TRUE will delete files after successful upload | FALSE | No |
| GRANULES\_UPLOAD\_TYPE | <p>Needs to be one of the following value</p><ul><li><code>UPLOAD_S3_BY_STAC_CATALOG</code> - for uploading files that have a STAC catalog that points to STAC Feature for each file</li><li><code>UPLOAD_AUXILIARY_FILE_AS_GRANULE</code> - for uploading a directory of files without providing STAC Catalog and STAC Features</li></ul> | UPLOAD\_S3\_BY\_STAC\_CATALOG | No |
| CATALOG\_FILE | | | Yes, if `GRANULES_UPLOAD_TYPE` is `UPLOAD_S3_BY_STAC_CATALOG` |
| BASE\_DIRECTORY | Path to directory containing files to upload | | Yes, if `GRANULES_UPLOAD_TYPE` is `UPLOAD_AUXILIARY_FILE_AS_GRANULE` |
| OUTPUT\_DIRECTORY | Path to directory to write successful and failed FeatureCollection | | Yes |
| LOG\_LEVEL | <p><code>10</code> = DEBUG</p><p><code>20</code> = INFO</p><p><code>30</code> = WARNING</p><p><code>40</code> = ERROR</p><p><code>50</code> = CRITICAL</p><p><code>60</code> = AUDIT</p> | 10 | No |
| PARALLEL\_COUNT | Number of threads used for uploading. Defaults to -1 which is equivalent to CPU count | -1 | No |

0 comments on commit 639d210

Please sign in to comment.