Bioethanol_Statistics_DynamoDB_AWS

Api Rest for the statistical management of production and sales of bioethanol based on cane and corn implemented with Api-Gateway, Nodemon, Serverless-Framework, NodeJs, DynamoDB, Systems Manager Parameter Store, Lambda among others. AWS services are tested locally. The project code and its documentation (less technical doc) have been developed in English.

• Bioethanol price reports
• Dataset biotenanol | National Data
• Excel Statistics Secretariat of Energy
• Playlist functionality test

Index 📜

View

Section 1) Description, configuration and technologies

• 1.0) Project Description.
• 1.1) Project Execution.
• 1.2) Project configuration from scratch
• 1.3) Technologies.

Section 2) Endpoints and Examples

• 2.0) EndPoints and resources.
• 2.1) Examples.

Section 3) Functionality Testing and References

• 3.0) Functionality test.
• 3.1) References.

Section 1) Description, configuration and technologies

1.0) Description 🔝

View

Api Rest for the statistical management of the production and marketing of bioethanol based on cane and corn. For its main architecture, all dynamo operations are covered through modularized helpers, endpoints through controllers, enumerations are used, etc. All necessary CRUD operations are also applied, as well as validations of credentials, tokens, headers, body, etc. for each endpoint of each table. The dynamodb tables involved are bioethanolPrices, bioethanolTotal, and bioethanolTypes. Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.

1.1) Project Execution 🔝

View

• We create a work environment through some IDE, we may or may not create a root folder for the project, we position ourselves on it

cd 'projectRootName'

• Once a work environment has been created, we clone the project

git clone https://github.com/andresWeitzel/Api_Bioetanol_Estadisticas_DynamoDB_AWS

• We position ourselves on the project

cd 'projectName'

• We install the latest LTS version of Nodejs(v18)
• We install the Serverless Framework globally if we have not already done so

npm install -g serverless

• We verify the version of Serverless installed

sls -v

• We install all the necessary packages

npm i

• Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.
• For simplification purposes, the file for ssm variables (serverless_ssm.yml) is included. It is recommended not to include or change credentials, token, etc.
• The following script configured in the project's package.json is responsible for
  • Lift serverless-offline ("serverless-offline")
  • run serverless-offline ("start")
  • run nodemon and serverless ("start:dev")
  • format all js and ts files with prettier ("format-prettier")
  • format all .md files with remark ("format-remark")
  • etc.

       "serverless-offline": "sls offline start",
       "start": "npm run serverless-offline",
       "start:dev": "nodemon -e js,ts,yml,json --exec \"sls offline start\"",
       "format-prettier": "prettier --write \"{src,test}/**/*.{js,ts}\"",
       "check": "remark . --quiet --frail",
       "format-remark": "remark . --quiet --frail --output",
       "format-md": "remark . --output"
  

  • We run the app from terminal.

  npm run start
  

  • We run the app with nodemon to auto detect changes from the server.

npm run start:dev

• Important: It is possible that there are other previous steps that have not been included due to synchronization between docs in relation to development. Please open a conversation thread within the 'Issues' section of the project.

1.2) Project configuration from scratch 🔝

View

• We create a work environment through some IDE, we may or may not create a root folder for the project, we position ourselves on it

cd 'projectRootName'

• Once a work environment has been created, we clone the project

git clone https://github.com/andresWeitzel/Api_Bioetanol_Estadisticas_DynamoDB_AWS

• We position ourselves on the project

cd 'projectName'

• We install the latest LTS version of Nodejs(v18)
• We install the Serverless Framework globally if we have not already done so

npm install -g serverless

• We verify the version of Serverless installed

sls -v

• We initialize a serverles template

serverless create --template aws-nodejs

• We initialize an npm project

npm init -y

• We install serverless offline

npm i serverless-offline --save-dev

• We add the plugin inside the serverless.yml

plugins:
   - serverless-offline

• We install serverless ssm

npm i serverless-offline-ssm --save-dev

• We add the plugin inside the serverless.yml

plugins:
   - serverless-offlline-ssm

• We install the plugin to use dynamodb locally (Not the dynamoDB service, this is configured in the files within .dynamodb).
• Important: There are security alerts from dependabot that were closed as they point to the "serverless-dynamodb-local" plugin. Do not apply security patches to that plugin, as version ^1.0.2 has problems creating tables and running the dynamo service. It is recommended to keep the latest stable version ^0.2.40 with the security alerts generated.

npm install serverless-dynamodb-local --save-dev

• We add the plugin inside the serverless.yml

plugins:
   - serverless-dynamodb-local

• We install the dynamodb client sdk for the necessary db operations

npm install @aws-sdk/client-dynamodb

• We install the dynamodb sdk lib for the necessary db operations

npm i @aws-sdk/lib-dynamodb

• We will modify the initial template for the standardized configs.
  • We replaced the initial serverless.yml template with the following one as the base model (change name, etc)...

service: name

frameworkVersion: '3'

provider:
   name: aws
   runtime: nodejs12.x
   stage: dev
   region: us-west-1
   memorySize: 512
   timeout: 10

plugins:
     - serverless-dynamodb-local
     - serverless-offline-ssm
     - serverless-offline

functions:
   Hello:
     handler: handler.hello

custom:
   serverless-offline:
     httpPort: 4000
     lambdaPort: 4002
   serverless-offline-ssm:
     stages:
       -dev
   dynamodb:
     stages:
       -dev

• We install prettier for indentations

npm i prettier --save

• We install node-input-validator to validate attributes in requests, class objects, etc.

npm i node-input-validator --save

• We must download the .jar along with its config to run the dynamodb service. Download here
• Once the .jar has been downloaded in .tar format, we decompress and copy all its contents into the .dynamodb folder.
• We install the dependency for the execution of scripts in parallel

npm i --save-dev concurrently

• The following script configured in the project's package.json is responsible for Raise serverless-offline (serverless-offline)

  "scripts": {
    "serverless-offline": "sls offline start",
    "start": "npm run serverless-offline"
  },

• We run the app from terminal.

npm start

• We should expect a console output with the following services raised when the previous command is executed

> crud-amazon-dynamodb-aws@1.0.0 start
> npm run serverless-offline

> crud-amazon-dynamodb-aws@1.0.0 serverless-offline
> sls offline start

serverless-offline-ssm checking serverless version 3.31.0.
Dynamodb Local Started, Visit: http://localhost:8000/shell
DynamoDB - created table xxxx

etc.....

• We already have a functional app with an initial structure defined by Serverless-Framework. The application is deployed at http://localhost:4002 and we can test the endpoint declared in the serverless from postman
• Clarification: The rest of the modifications applied to the initial template are not described due to document simplification issues. For more information consult See the Serverless-framework tutorial for using services, plugins, etc.

1.3) Technologies 🔝

View

Technologies	Version	Purpose
SDK	4.3.2	Automatic Module Injection for Lambdas
Serverless Framework Core v3	3.23.0	Core Services AWS
Serverless Plugin	6.2.2	Libraries for Modular Definition
Systems Manager Parameter Store (SSM)	3.0	Management of Environment Variables
Amazon Api Gateway	2.0	API Manager, Authentication, Control and Processing
Amazon DynamoDB	2017.11.29	Fast and flexible NoSQL database service for single-digit millisecond performance at any scale
NodeJS	14.18.1	JS Library
VSC	1.72.2	IDE
Postman	10.11	Http Client
CMD	10	Command Prompt for command line
Git	2.29.1	Version Control
Others	-	Others

Plugin	Description
Serverless Plugin	Libraries for Modular Definition
serverless-dynamodb-local	Allows to run dynamodb locally for serverless
serverless-offline	This serverless plugin emulates AWS λ and API Gateway on-premises
serverless-offline-ssm	finds environment variables that match the SSM parameters at build time and replaces them from a file

VSC Extensions Implemented.

Extension
Prettier - Code formatter
YAML - Autoformatter .yml (alt+shift+f)
GitLens - Tracking changes
Serverless Framework - Autocompleted with snippets
Tabnine - AI Autocomplete
Others

Section 2) Endpoints and Examples.

2.0) Endpoints and resources 🔝

View

2.0.1) Variables in Postman

Variable	Value
base_url	http://localhost:4000/dev/v1
x-api-key	f98d8cd98h73s204e3456998ecl9427j
bearer-token	Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

• Important: Key values included are for local testing only.

2.0.2) Bioetanol_Precios endpoints

GET type operations:

• base_url/bioetanol-precios/list?limit=limitValue&orderAt=orderAtValue
• base_url/bioetanol-precios/uuid/uuidValue
• base_url/bioetanol-precios/bioetanol-cana-azucar/bioetanolCanaAzucarValue?limit=limitValue&orderAt=orderAtValue
• base_url/bioetanol-precios/created-at/createdAtvalue?limit=limitValue&orderAt=orderAtValue
• base_url/bioetanol-precios/field-type?limit=limitValue&orderAt=orderAtValue&fieldType=fieldTypeValue&fieldValue=fieldValueValue
• base_url/bioetanol-precios/periodo/periodoValue
• base_url/bioetanol-precios/bioetanol-maiz/bioetanolMaizValue?limit=limitValue&orderAt=orderAtValue
• All endpoints are optional paginated except /test, /db-connection and /id/{{user-id}}

POST type operations:

• base_url/bioetanol-precios/

PUT type operations:

• base_url/bioetanol-precios/uuid

DELETE type operations:

• base_url/bioetanol-precios/uuid

2.0.3) Bioetanol_Tipos endpoints

• To summarize the documentation, review the postman collection endpoints

2.0.4) Bioetanol_Total endpoints

• To summarize the documentation, review the postman collection endpoints

2.1) Examples 🔝

View

2.1.0) Variables in Postman

Variable	Value
base_url	http://localhost:4000/dev/v1
x-api-key	f98d8cd98h73s204e3456998ecl9427j
bearer-token	Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

• Important: Key values included are for local testing only.

2.1.1) Bioetanol_Precios endpoints

Get All Bioetanol-precios items

Request (GET)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/list?limit=3&orderAt=asc' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json'

Response (200 OK)

{
    "message": [
        {
            "createdAt": "2023-11-18 21:55:01",
            "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
            "periodo": "2023-11",
            "bioetMaiz": "412,23",
            "bioetCanAzucar": "345,33",
            "updatedAt": "2023-11-18 21:55:01"
        }
    ]
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "An error has occurred, failed to list database objects. Check if items exists."
}

Get By UUID Bioetanol-precios

Request (GET)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/uuid/3f86f08e-99a6-442f-b31c-1668cbe76edb' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json'

Response (200 OK)

{
    "message": {
        "createdAt": "2023-11-18 21:55:01",
        "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
        "periodo": "2023-11",
        "bioetMaiz": "412,23",
        "bioetCanAzucar": "345,33",
        "updatedAt": "2023-11-18 21:55:01"
    }
}

Response (400 Bad Request)

{
    "message": "The Bioetanol prices object with the requested id 3f86f08e-99a6-442f-b31c-1668cbe76edb is not found in the database."
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

• To summarize the documentation, review the postman collection endpoints for GET operations.

Add Bioetanol-precios item

Request (POST)

curl --location 'http://localhost:4000/dev/v1/bioetanol-precios/' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
    "periodo":"2023-11",
    "bioetanol_azucar":"345,33",
    "bioetanol_maiz":"412,23"
}'

Response (200 OK)

{
    "message": {
        "uuid": "3bfff0ca-8cba-4113-bc94-4afb6e7feb7e",
        "periodo": "2023-11",
        "bioetCanAzucar": "345,33",
        "bioetMaiz": "412,23",
        "createdAt": "2023-11-18 21:55:01",
        "updatedAt": "2023-11-18 21:55:01"
    }
}

Response (400 Bad Request)

{
    "message": "Bad request, check request body attributes. Missing or incorrect"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Update Bioetanol-precios item

Request (PUT)

curl --location --request PUT 'http://localhost:4000/dev/v1/bioetanol-precios/67ecfcf7-c338-43d8-9220-4d7b43b7e914' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
    "periodo":"2023-11",
    "bioetanol_azucar":"345,33",
    "bioetanol_maiz":"412,23"
}'

Response (200 OK)

{
    "message": {
        "createdAt": "2023-11-18 22:01:34",
        "periodo": "2023-11",
        "uuid": "b58fd5cb-ed0b-461c-bfea-50c240e51280",
        "bioetMaiz": "412,23",
        "bioetCanAzucar": "345,33",
        "updatedAt": "2023-11-18 22:03:34"
    }
}

Response (400 Bad Request)

{
    "message": "Bad request, check request body attributes for bioetanol-precios. Missing or incorrect"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "Internal Server Error. Unable to update object in db as failed to get a item by uuid 67ecfcf7-c338-43d8-9220-4d7b43b7e914 . Check if the item exists in the database and try again."
}

Delete Bioetanol-precios item

Request (DELETE)

curl --location --request DELETE 'http://localhost:4000/dev/v1/bioetanol-precios/2c6d2e51-390b-4cb4-ab69-7820c632e6a4' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data ''

Response (200 OK)

{
    "message": "Successfully removed item based on uuid b58fd5cb-ed0b-461c-bfea-50c240e51280"
}

Response (400 Bad Request)

{
    "message": "Bad request, check missing or malformed headers"
}

Response (401 Unauthorized)

{
    "message": "Not authenticated, check x_api_key and Authorization"
}

Response (500 Internal Server Error)

{
    "message": "Unable to delete item based on uuid 2c6d2e51-390b-4cb4-ab69-7820c632e6a4"
}

Section 3) Functionality Testing and References.

3.0) Functionality test 🔝

View

3.1) References 🔝

View

Dynamodb installation

• DynamoDB on local executable

DynamoDB theory

• DynamoDB Guide
• Official Api DynamoDB Doc
• Attribute definition
• Partition Key vs Sort
• Filter Expressions in Dynamodb
• Examples of Filter Expressions in Dynamodb

Dynamodb operations sdk v-3

• Operations
• Operations API-REST

Video tutorials

• Dynamodb local config
• Crud Dynamodb

Dynamodb examples

• serverless plugin
• Creating multiple tables
• dynamodb serverless example
• Dynamodb SDK examples
• CRUD Dynamodb

Dynamodb code

• Api Rest Base

Tools

• AWS Design Tool app.diagrams.net

API Gateway

• Hello good Api-Gateway Practices
• Creating Custom Api-keys

Bookstores

• Field validation
• uuidv4 generator
• Nodemon Usage

Package.json

• Setting up parallel scripts

Formating prettier

• Formatting Node.js codebase with Prettier
• Set up a Node.js App with ESLint and Prettier

Formating remark-link

• remark-inline-links
• remark-lint-list-item-indent

Testing

• How to mock process env in jest