DATA CRUNCH ENGINE

*** THE REPO IS UNDER CONSTRUCTION - CHECK BACK SOON ***

A lightweight asynchronous microservice that will receive data, crunch data and return the results. You provide the data crunching algorithm.

Table of Contents

• OVERVIEW
• PREREQUISITES
• SOFTWARE STACK
• PROTOCOL COMPILE FOR GO
• RUN
• CREATE BINARY
• STEP 1 - TEST
• STEP 2 - BUILD (DOCKER IMAGE VIA DOCKERFILE)
• STEP 3 - PUSH (TO DOCKERHUB)
• STEP 4 - DEPLOY (TO DOCKER)
• CONTINUOUS INTEGRATION & DEPLOYMENT

Documentation and References

• The data-crunch-engine docker image on DockerHub
• This repos github webpage built with concourse

OVERVIEW

This data-crunch-engine is,

• Written in go
• Utilizes goroutines (concurrency)
• Uses protobuf over NATS for messaging
• Built to a lightweight Docker Image

This illustration shows a high level view,

Notice that you may have multiple data-crunch-engines running.

And a more detailed view of the data-crunch engine,

PREREQUISITES

You will need the following go packages,

go get -u -v github.com/sirupsen/logrus
go get -u -v github.com/cweill/gotests/...

SOFTWARE STACK

• DEVELOPMENT
  • go
  • gotests
• OPERATIONS
  • concourse/fly (optional)
  • docker
• SERVICES
  • dockerhub
  • github

Where,

• GUI golang net/http package and ReactJS
• Routing & REST API framework golang gorilla/mux package
• Backend golang
• Database N/A

PROTOCOL COMPILE FOR GO

The protocol buffer human readable file is located here

The two interfaces have been defined as,

// Check your error
func checkErr(err error) {
    if err != nil {
        log.Fatal("ERROR:", err)
    }
}

message MyData {
    int64 ID = 1;
    int64 Data = 2;
    string Meta = 3;
}

message MyResult {
    int64 ID = 1;
    int64 Data = 2;
    string Meta = 3;
    google.protobuf.Timestamp DTimeStamp = 4;
    int64 RData = 5;
    google.protobuf.Timestamp ProcessTime = 6;
}

This file has already been compiled, but you may recompile it using the shell script.

RUN

First, start your NATS server,

nats-server -DV -p 4222 -a 127.0.0.1

In separate terminals start the data-engine, the data-crunch-engine and the results-engine respectively,

go run data-engine.go messages.pb.go
go run data-crunch-engine.go messages.pb.go
go run results-engine.go messages.pb.go

Currently, I have a placeholder as follows,

To run.sh,

cd data-crunch-engine-code
go run main.go

As a placeholder, every 2 seconds it will print,

    INFO[0000] Let's Start this!
    Hello everyone, count is: 1
    Hello everyone, count is: 2
    Hello everyone, count is: 3
    etc...

CREATE BINARY

To create-binary.sh,

cd data-crunch-engine-code/bin
go build -o data-crunch-engine ../main.go
./data-crunch-engine

This binary will not be used during a docker build since it creates it's own.

STEP 1 - TEST

To create unit _test files,

cd data-crunch-engine-code
gotests -w -all main.go

To run unit-tests.sh,

go test -cover ./... | tee test/test_coverage.txt
cat test/test_coverage.txt

STEP 2 - BUILD (DOCKER IMAGE VIA DOCKERFILE)

To build.sh with a Dockerfile,

cd data-crunch-engine-code
docker build -f build/Dockerfile -t jeffdecola/data-crunch-engine .

You can check and test this docker image,

docker images jeffdecola/data-crunch-engine:latest
docker run --name data-crunch-engine -dit jeffdecola/data-crunch-engine
docker exec -i -t data-crunch-engine /bin/bash
docker logs data-crunch-engine
docker rm -f data-crunch-engine

In stage 1, rather than copy a binary into a docker image (because that can cause issues), the Dockerfile will build the binary in the docker image,

FROM golang:alpine AS builder
RUN go get -d -v
RUN go build -o /go/bin/data-crunch-engine main.go

In stage 2, the Dockerfile will copy the binary created in stage 1 and place into a smaller docker base image based on alpine, which is around 13MB.

STEP 3 - PUSH (TO DOCKERHUB)

You must be logged in to DockerHub,

docker login

To push.sh,

docker push jeffdecola/data-crunch-engine

Check the data-crunch-engine docker image at DockerHub.

STEP 4 - DEPLOY (TO DOCKER)

To deploy.sh,

cd data-crunch-engine-code
docker run --name data-crunch-engine -dit jeffdecola/data-crunch-engine
docker exec -i -t data-crunch-engine /bin/bash
docker logs data-crunch-engine
docker rm -f data-crunch-engine

CONTINUOUS INTEGRATION & DEPLOYMENT

Refer to ci-README.md on how I automated the above steps.