ylem-rabbitmq-consumer

Golang-based RabbitMQ consumer to consume messages from your queues, forward messages to Ylem pipelines, and trigger them in real-time.

Usage

ENV variables

Create the new .env and copy the content from .env.dist to it and then add your values to the list of parameters:

• AMQP_SERVER_URL - the URL or your Rabbit MQ server in the format of: amqp://guest:guest@ylem-message-broker:5672/
• YLEM_API_URL - If you use the cloud version of Ylem, it is https://api.ylem.co, othervise add URL of your custom Ylem API instance here
• YLEM_API_CLIENT_ID - API Client ID. Here is how to create it: https://docs.datamin.io/datamin-api/oauth-clients
• YLEM_API_CLIENT_SECRET - API Client Secret. Here is how to create it: https://docs.datamin.io/datamin-api/oauth-clients
• QUEUE_NAME - Queue name, the consumer should listen to
• PIPELINE_UUID - UUID of the pipeline to trigger in case of the new messages in the queue

For example:

AMQP_SERVER_URL=amqp://guest:guest@ylem-message-broker:5672/ 
YLEM_API_URL=https://api.ylem.co
YLEM_API_CLIENT_ID=be48d317-924c-4b1c-809a-026638e447b7
YLEM_API_CLIENT_SECRET=cfa9dd5e33cd8b2f0f604d94b5gggipgyggggguje2cefb
QUEUE_NAME=ylem_queue
PIPELINE_UUID=2c334d54-c807-4629-99eb-a4def2455557

Run the service

make run

or in the debug mode:

docker-compose up --build

By default it launches three docker containers (see docker-compose.yml):

• Message broker
• Producer
• Consumer

The first two are optional and are not needed in case you already have your own RabbitMQ service and data producers that populate queues.

In this case you can run the consumer container only:

docker run -itd --rm \
  -e AMQP_SERVER_URL='%YOUR_AMQP_SERVER_URL%' \
  -e YLEM_API_URL='%YLEM_API_URL%' \
  -e YLEM_API_CLIENT_ID='%YLEM_API_CLIENT_ID%' \
  -e YLEM_API_CLIENT_SECRET='%YLEM_API_CLIENT_SECRET%' \
  -e QUEUE_NAME='%QUEUE_NAME%' \
  -e PIPELINE_UUID='%PIPELINE_UUID%' \
	--network=ylem-rabbitmq-consumer_ylem-rmq-network   \
	ylem-rabbitmq-consumer-ylem-consumer

Using the same command you can run more containers with consumers listening to the same or other queues.

GUI

RabbitMQ administrative interface is available here: http://localhost:15672/

Test flow

1. Create the pipeline

As an example, we will use a simple pipeline that receives a message from RabbitMQ and sends an Email notification:

Notification task:

2. Send test message to the producer API:

API Endpoint: http://localhost:3240/message

API Endpoint expects the POST request with the JSON body

{
    "info": "some information", // just an example
    "customer_id": 1 // just an example
}

Valid input

Invalid input. Validation error is returned

3. Check CLI output to make sure it was received by the producer and consumed

4. Check you Email inbox and confirm that the message is received

Full documentation

https://docs.datamin.io/

Development

Linter

Install Golang linter on MacOS

$ brew install golangci-lint
$ brew upgrade golangci-lint

Check the code with it

$ golangci-lint run

More information is in the official documentation: https://golangci-lint.run/