Brazilian E-Commerce ETL

Simple application in NestJS to perform ETL in the construction of a Data Warehouse on e-commerce sales in Brazil in the period 2016-2018.

About The Project

Project of the discipline Projeto Integrador (Integrator Project) of the 3º semester of the course of Technology in Big Data for Business at FATEC Ipiranga. Supervised by Antonio Guardado.

Portuguese description

A ideia do projeto é fazer a realização de análises sobre uma determinada base de dados e fazer uma apresentação visual dos resultados encontrados.

Foi selecionado a utilização do dataset Brazilian E-Commerce Public Dataset by Olist, que possui dados históricos de e-commerce no Brasil, no período de 2016 até 2018.

Built With

• Node v16.13.1
• Typescript 4.7.2
• NestJS
• Prisma

Getting Started

For the use of the project, some prerequisites will be necessary.

Prerequisites

• Node.JS

  1. You can download here: Node.JS
  2. Here is a step-by-step installation tutorial. (Windows) (Linux)

• PostgreSQL (optional, if you are going to use Docker)

  1. You can download here: PostgreSQL
  2. Here is a step-by-step installation tutorial. (Windows) (Linux)

• npx

  npm install -g npx

Installation

1. Clone the repository

   git clone https://github.com/Arekushi/brazilian-e-commerce-js.git

2. Install NPM packages

   npm install

3. Create .env at project root and add this

   DATABASE_URL=postgresql://[user]:[password]@localhost:5432/brazilian_e_commerce

4. Run in terminal

   npm run command init:main

5. All ready, the datasets will be downloaded and the inclusion in the database will be done 🎉

Roadmap

• Creating models in Prisma
• Reading module
  • Abstract base class
  • Read the CSV
  • Perform type transformations (Integer, Float, Date, Boolean)
• Writing module
  • Abstract base class
  • Mapping the database model with the CSV interface
• CLI module
  • Creating commands for each CSV file
  • Creating a command to download the datasets (CSV files)
    • Download file request
    • Unzip module
• Test

Schema

Here is a dictionary of data in a simple Notion document about entities, their attributes and relationships. (Schema)

Acknowledgments

• O que é DATA WAREHOUSE? Você precisa de um?

Contributors

Alexandre Ferreira de Lima	Eduardo Germano de Oliveira	Gilberto Magalhães	Mychelle Roberto Veloso	Diego Silva Neves	Ana Carolina Ferreira de Camargo