Skip to content

rtista/Gatherer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

81 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Gatherer - Data Collection Web Application

HTTP API allows sending raw JSON formatted data into an ActiveMQ queue for later custom consumption. Consumers should be developed for each queue as each should save different data in a different way.

System Dependencies

First you may need to install python dependencies on your system.

The table below shows which technologies require the packages whose installation is required.

Package Required By
libev-devel Bjoern HTTP Server
python2-devel Bjoern HTTP Server
libpq-devel Postgres SQL Driver (psycopg2)
mariadb-devel MariaDB/MySQL Driver (mysqlclient)

On RedHat Based Distributions - Dnf

sudo dnf install libev-devel python3-devel libpq-devel mariadb-devel

On RedHat Based Distributions - Yum

sudo yum install libev-devel python3-devel libpq-devel mariadb-devel

On Debian Based Distributions - Apt

sudo apt-get install libev-devel python3-devel libpq-dev mariadb-dev

Cloning

git clone git@github.com:rtista/Gatherer.git

Pip Dependencies

cd Gatherer/
pip install -r requirements.txt

Testing with dockers

You can test the application using dockers to provide architecture required systems (i.e. MariaDB, ActiveMQ, etc):

ActiveMQ

The following command will provide an ActiveMQ running on 127.0.0.1.

docker run --name activemq -p 8161:8161 -p 1883:1883 -p 5672:5672 -p 61613:61613 -p 61614:61614 -p 61616:61616 -d activemq

ActiveMQ provides a management graphical web interface which you can access onn http://127.0.0.1:8161/admin. In order to queue and dequeue messages you may use any of the following protocols:

Port Protocol Description
1883 MQTT Message Queuing Telemetry Transport
5672 AMQP ActiveMQ Messaging Protocol
61613 Stomp Simple/Streaming Text Oriented Message Protocol
61614 WS WebSocket
61616 OpenWire ActiveMQ Native Protocol

MariaDB

Run the following command and you may access the database on 127.0.0.1:3306 with the 'root' user and 'password' password.

docker run --name mariadb -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mariadb

Postgres

Run the following command and you may access the database on 127.0.0.1:5324 with the 'postgres' user and no password at the 'postgres' database.

docker run --name postgres -p 5432:5432 -d postgres