Bioethanol_Statistics_DynamoDB_AWS

Api Rest for the statistical management of production and sales of bioethanol based on cane and corn implemented with Api-Gateway, Nodemon, Serverless-Framework, NodeJs, DynamoDB, Systems Manager Parameter Store, Lambda among others. AWS services are tested locally. The project code and its documentation (less technical doc) have been developed in English.

Bioethanol price reports
Dataset biotenanol | National Data
Excel Statistics Secretariat of Energy
Playlist functionality test

Index 📜

View

Section 1) Description, configuration and technologies

1.0) Project Description.
1.1) Project Execution.
1.2) Project configuration from scratch
1.3) Technologies.

Section 2) Endpoints and Examples

2.0) EndPoints and resources.
2.1) Examples.

Section 3) Functionality Testing and References

3.0) Functionality test.
3.1) References.

Section 1) Description, configuration and technologies

1.0) Description 🔝

View

Api Rest for the statistical management of the production and marketing of bioethanol based on cane and corn. For its main architecture, all dynamo operations are covered through modularized helpers, endpoints through controllers, enumerations are used, etc. All necessary CRUD operations are also applied, as well as validations of credentials, tokens, headers, body, etc. for each endpoint of each table. The dynamodb tables involved are bioethanolPrices, bioethanolTotal, and bioethanolTypes. Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.

1.1) Project Execution 🔝

View

We create a work environment through some IDE, we may or may not create a root folder for the project, we position ourselves on it

cd 'projectRootName'

Once a work environment has been created, we clone the project

git clone https://github.com/andresWeitzel/Api_Bioetanol_Estadisticas_DynamoDB_AWS

We position ourselves on the project

cd 'projectName'

We install the latest LTS version of Nodejs(v18)
We install the Serverless Framework globally if we have not already done so

npm install -g serverless

We verify the version of Serverless installed

sls -v

We install all the necessary packages

npm i

Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.
For simplification purposes, the file for ssm variables (serverless_ssm.yml) is included. It is recommended not to include or change credentials, token, etc.
The following script configured in the project's package.json is responsible for
- Lift serverless-offline ("serverless-offline")
- run serverless-offline ("start")
- run nodemon and serverless ("start:dev")
- format all js and ts files with prettier ("format-prettier")
- format all .md files with remark ("format-remark")
- etc.
```
     "serverless-offline": "sls offline start",
     "start": "npm run serverless-offline",
     "start:dev": "nodemon -e js,ts,yml,json --exec \"sls offline start\"",
     "format-prettier": "prettier --write \"{src,test}/**/*.{js,ts}\"",
     "check": "remark . --quiet --frail",
     "format-remark": "remark . --quiet --frail --output",
     "format-md": "remark . --output"
```
- We run the app from terminal.
```
npm run start
```
- We run the app with nodemon to auto detect changes from the server.

npm run start:dev

Important: It is possible that there are other previous steps that have not been included due to synchronization between docs in relation to development. Please open a conversation thread within the 'Issues' section of the project.

1.2) Project configuration from scratch 🔝

View

We create a work environment through some IDE, we may or may not create a root folder for the project, we position ourselves on it

cd 'projectRootName'

Once a work environment has been created, we clone the project

git clone https://github.com/andresWeitzel/Api_Bioetanol_Estadisticas_DynamoDB_AWS

We position ourselves on the project

cd 'projectName'

We install the latest LTS version of Nodejs(v18)
We install the Serverless Framework globally if we have not already done so

npm install -g serverless

We verify the version of Serverless installed

sls -v

We initialize a serverles template

serverless create --template aws-nodejs

We initialize an npm project

npm init -y

We install serverless offline

npm i serverless-offline --save-dev

We add the plugin inside the serverless.yml

plugins:
   - serverless-offline

We install serverless ssm

npm i serverless-offline-ssm --save-dev

We add the plugin inside the serverless.yml

plugins:
   - serverless-offlline-ssm

We install the plugin to use dynamodb locally (Not the dynamoDB service, this is configured in the files within .dynamodb).
Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.

npm install serverless-dynamodb-local --save-dev

We add the plugin inside the serverless.yml

plugins:
   - serverless-dynamodb-local

We install the dynamodb client sdk for the necessary db operations

npm install @aws-sdk/client-dynamodb

We install the dynamodb sdk lib for the necessary db operations

npm i @aws-sdk/lib-dynamodb

We will modify the initial template for the standardized configs.
- We replaced the initial serverless.yml template with the following one as the base model (change name, etc)...

service: name

frameworkVersion: '3'

provider:
   name: aws
   runtime: nodejs12.x
   stage: dev
   region: us-west-1
   memorySize: 512
   timeout: 10

plugins:
     - serverless-dynamodb-local
     - serverless-offline-ssm
     - serverless-offline

functions:
   Hello:
     handler: handler.hello

custom:
   serverless-offline:
     httpPort: 4000
     lambdaPort: 4002
   serverless-offline-ssm:
     stages:
       -dev
   dynamodb:
     stages:
       -dev

We install prettier for indentations

npm i prettier --save

We install node-input-validator to validate attributes in requests, class objects, etc.

npm i node-input-validator --save

We must download the .jar along with its config to run the dynamodb service. Download here
Once the .jar has been downloaded in .tar format, we decompress and copy all its contents into the .dynamodb folder.
We install the dependency for the execution of scripts in parallel

npm i --save-dev concurrently

The following script configured in the project's package.json is responsible for Raise serverless-offline (serverless-offline)

  "scripts": {
    "serverless-offline": "sls offline start",
    "start": "npm run serverless-offline"
  },

We run the app from terminal.

npm start

We should expect a console output with the following services raised when the previous command is executed

> crud-amazon-dynamodb-aws@1.0.0 start
> npm run serverless-offline

> crud-amazon-dynamodb-aws@1.0.0 serverless-offline
> sls offline start

serverless-offline-ssm checking serverless version 3.31.0.
Dynamodb Local Started, Visit: http://localhost:8000/shell
DynamoDB - created table xxxx

etc.....

We already have a functional app with an initial structure defined by Serverless-Framework. The application is deployed at http://localhost:4002 and we can test the endpoint declared in the serverless from postman
Clarification: The rest of the modifications applied to the initial template are not described due to document simplification issues. For more information consult See the Serverless-framework tutorial for using services, plugins, etc.

1.3) Technologies 🔝

View

Technologies	Version	Purpose
SDK	4.3.2	Automatic Module Injection for Lambdas
Serverless Framework Core v3	3.23.0	Core Services AWS
Serverless Plugin	6.2.2	Libraries for Modular Definition
Systems Manager Parameter Store (SSM)	3.0	Management of Environment Variables
Amazon Api Gateway	2.0	API Manager, Authentication, Control and Processing
Amazon DynamoDB	2017.11.29	Fast and flexible NoSQL database service for single-digit millisecond performance at any scale
NodeJS	14.18.1	JS Library
VSC	1.72.2	IDE
Postman	10.11	Http Client
CMD	10	Command Prompt for command line
Git	2.29.1	Version Control
Others	-	Others

Plugin	Description
Serverless Plugin	Libraries for Modular Definition
serverless-dynamodb-local	Allows to run dynamodb locally for serverless
serverless-offline	This serverless plugin emulates AWS λ and API Gateway on-premises
serverless-offline-ssm	finds environment variables that match the SSM parameters at build time and replaces them from a file

VSC Extensions Implemented.

Extension
Prettier - Code formatter
YAML - Autoformatter .yml (alt+shift+f)
GitLens - Tracking changes
Serverless Framework - Autocompleted with snippets
Tabnine - AI Autocomplete
Others

Section 2) Endpoints and Examples.

2.0) Endpoints and resources 🔝

View

2.0.1) Variables in Postman

Variable	Value
base_url	http://localhost:4000/dev/v1
x-api-key	f98d8cd98h73s204e3456998ecl9427j
bearer-token	Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Important: Key values included are for local testing only.

2.0.2) Bioetanol_Precios endpoints

GET type operations:

base_url/bioetanol-precios/list?limit=limitValue&orderAt=orderAtValue
base_url/bioetanol-precios/uuid/uuidValue
base_url/bioetanol-precios/bioetanol-cana-azucar/bioetanolCanaAzucarValue?limit=limitValue&orderAt=orderAtValue
base_url/bioetanol-precios/created-at/createdAtvalue?limit=limitValue&orderAt=orderAtValue
base_url/bioetanol-precios/field-type?limit=limitValue&orderAt=orderAtValue&fieldType=fieldTypeValue&fieldValue=fieldValueValue
base_url/bioetanol-precios/periodo/periodoValue
base_url/bioetanol-precios/bioetanol-maiz/bioetanolMaizValue?limit=limitValue&orderAt=orderAtValue
All endpoints are optional paginated except /test, /db-connection and /id/{{user-id}}

POST type operations:

base_url/bioetanol-precios/

PUT type operations:

base_url/bioetanol-precios/uuid

DELETE type operations:

base_url/bioetanol-precios/uuid

2.0.3) Bioetanol_Tipos endpoints

To summarize the documentation, review the postman collection endpoints

2.0.4) Bioetanol_Total endpoints

To summarize the documentation, review the postman collection endpoints

2.1) Examples 🔝

View

2.1.0) Variables in Postman

Variable	Value
base_url	http://localhost:4000/dev/v1
x-api-key	f98d8cd98h73s204e3456998ecl9427j
bearer-token	Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Important: Key values included are for local testing only.

2.1.1) Bioetanol_Precios endpoints

Get All Bioetanol-precios items

Request (GET)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/list?limit=3&orderAt=asc' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json'

Response (200 OK)

{
    "message": [
        {
            "createdAt": "2023-11-18 21:55:01",
            "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
            "periodo": "2023-11",
            "bioetMaiz": "412,23",
            "bioetCanAzucar": "345,33",
            "updatedAt": "2023-11-18 21:55:01"
        }
    ]
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "An error has occurred, failed to list database objects. Check if items exists."
}

Get By UUID Bioetanol-precios

Request (GET)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/uuid/3f86f08e-99a6-442f-b31c-1668cbe76edb' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json'

Response (200 OK)

{
    "message": {
        "createdAt": "2023-11-18 21:55:01",
        "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
        "periodo": "2023-11",
        "bioetMaiz": "412,23",
        "bioetCanAzucar": "345,33",
        "updatedAt": "2023-11-18 21:55:01"
    }
}

Response (400 Bad Request)

{
    "message": "The Bioetanol prices object with the requested id 3f86f08e-99a6-442f-b31c-1668cbe76edb is not found in the database."
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

To summarize the documentation, review the postman collection endpoints for GET operations.

Add Bioetanol-precios item

Request (POST)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
    "periodo":"2023-11",
    "bioetanol_azucar":"345,33",
    "bioetanol_maiz":"412,23"
}'

Response (200 OK)

{
    "message": {
        "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
        "periodo": "2023-11",
        "bioetCanAzucar": "345,33",
        "bioetMaiz": "412,23",
        "createdAt": "2023-11-18 21:55:01",
        "updatedAt": "2023-11-18 21:55:01"
    }
}

Response (400 Bad Request)

{
    "message": "Bad request, check request body attributes. Missing or incorrect"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Update Bioetanol-precios item

Request (PUT)

curl --location --request PUT 'http://localhost:4000/dev/v1/bioetanol-precios/67ecfcf7-c338-43d8-9220-4d7b43b7e914' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
    "periodo":"2023-11",
    "bioetanol_azucar":"345,33",
    "bioetanol_maiz":"412,23"
}'

Response (200 OK)

{
    "message": {
        "createdAt": "2023-11-18 22:01:34",
        "periodo": "2023-11",
        "uuid": "b58fd5cb-ed0b-461c-bfea-50c240e51280",
        "bioetMaiz": "412,23",
        "bioetCanAzucar": "345,33",
        "updatedAt": "2023-11-18 22:03:34"
    }
}

Response (400 Bad Request)

{
    "message": "Bad request, check request body attributes for bioetanol-precios. Missing or incorrect"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "Internal Server Error. Unable to update object in db as failed to get a item by uuid 67ecfcf7-c338-43d8-9220-4d7b43b7e914 . Check if the item exists in the database and try again."
}

Delete Bioetanol-precios item

Request (DELETE)

curl --location --request DELETE 'http://localhost:4000/dev/v1/bioetanol-precios/2c6d2e51-390b-4cb4-ab69-7820c632e6a4' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data ''

Response (200 OK)

{
    "message": "Successfully removed item based on uuid b58fd5cb-ed0b-461c-bfea-50c240e51280"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "Unable to delete item based on uuid 2c6d2e51-390b-4cb4-ab69-7820c632e6a4"
}

Section 3) Functionality Testing and References.

3.0) Functionality test 🔝

View

3.1) References 🔝

View

Dynamodb installation

DynamoDB on local executable

DynamoDB theory

DynamoDB Guide
Official Api DynamoDB Doc
Attribute definition
Partition Key vs Sort
Filter Expressions in Dynamodb
Examples of Filter Expressions in Dynamodb

Dynamodb operations sdk v-3

Operations
Operations API-REST

Video tutorials

Dynamodb local config
Crud Dynamodb

Dynamodb examples

serverless plugin
Creating multiple tables
dynamodb serverless example
Dynamodb SDK examples
CRUD Dynamodb

Dynamodb code

Api Rest Base

Tools

AWS Design Tool app.diagrams.net

API Gateway

Hello good Api-Gateway Practices
Creating Custom Api-keys

Bookstores

Field validation
uuidv4 generator
Nodemon Usage

Package.json

Setting up parallel scripts

Formating prettier

Formatting Node.js codebase with Prettier
Set up a Node.js App with ESLint and Prettier

Formating remark-link

remark-inline-links
remark-lint-list-item-indent

Testing

How to mock process env in jest

Files

README.md

Latest commit

History

README.md

File metadata and controls

Bioethanol_Statistics_DynamoDB_AWS

Index 📜

Section 1) Description, configuration and technologies

Section 2) Endpoints and Examples

Section 3) Functionality Testing and References

Section 1) Description, configuration and technologies

1.0) Description 🔝

1.1) Project Execution 🔝

1.2) Project configuration from scratch 🔝

1.3) Technologies 🔝

VSC Extensions Implemented.

Section 2) Endpoints and Examples.

2.0) Endpoints and resources 🔝

2.0.1) Variables in Postman

2.0.2) Bioetanol_Precios endpoints

GET type operations:

POST type operations:

PUT type operations:

DELETE type operations:

2.0.3) Bioetanol_Tipos endpoints

2.0.4) Bioetanol_Total endpoints

2.1) Examples 🔝

2.1.0) Variables in Postman

2.1.1) Bioetanol_Precios endpoints

Get All Bioetanol-precios items

Request (GET)

Response (200 OK)

Response (400 Bad Request)

Response (401 Unauthorized)

Response (500 Internal Server Error)

Get By UUID Bioetanol-precios

Request (GET)

Response (200 OK)

Response (400 Bad Request)

Response (400 Bad Request)

Response (401 Unauthorized)

Add Bioetanol-precios item

Request (POST)

Response (200 OK)

Response (400 Bad Request)

Response (400 Bad Request)

Response (401 Unauthorized)

Update Bioetanol-precios item

Request (PUT)

Response (200 OK)

Response (400 Bad Request)

Response (400 Bad Request)

Response (401 Unauthorized)

Response (500 Internal Server Error)

Delete Bioetanol-precios item

Request (DELETE)

Response (200 OK)

Response (400 Bad Request)

Response (401 Unauthorized)

Response (500 Internal Server Error)

Section 3) Functionality Testing and References.

3.0) Functionality test 🔝

3.1) References 🔝

Dynamodb installation

DynamoDB theory

Dynamodb operations sdk v-3

Video tutorials

Dynamodb examples

Dynamodb code

Tools

API Gateway

Bookstores

Package.json

Formating prettier

Formating remark-link

Testing