NestJS + Graphile Worker App

A progressive Node.js framework integrated with Graphile Worker to create a robust and scalable queue processing system.

---

Overview

This project demonstrates how to use NestJS with Graphile Worker to build a queue-based processing system. It leverages the scalability and modularity of NestJS alongside the powerful job queueing capabilities of Graphile Worker.

Features

• NestJS Framework: A progressive Node.js framework for building efficient server-side applications.
• Graphile Worker Integration: A lightweight and high-performance job queue for PostgreSQL.
• Queue Management: Easily define and process tasks in a queue.
• Scalability: Designed to handle high-throughput workloads.

Prerequisites

Before setting up the project, ensure you have the following installed:

• Node.js (v16 or higher recommended)
• PostgreSQL (v12 or higher)
• npm or yarn

Installation

Clone the repository and install dependencies:

$ git clone https://github.com/fernando-moro/nestjs-graphile-worker-app.git
$ cd nestjs-graphile-worker-app
$ npm install

Configuration

1. Create a .env file in the root directory and configure your PostgreSQL connection:

POSTGRESQL_URI=postgres://username:password@localhost:5432/your_database

2. Database migrations will run automatically when the application starts.

Running the Application

Start the application in different modes:

# Development mode
$ npm run start

# Watch mode (auto-restart on file changes)
$ npm run start:dev

# Production mode
$ npm run start:prod

Using the Queue

1. Define a task in the tasks folder.
2. Add jobs to the queue using the Graphile Worker API.
3. Process jobs automatically with the worker.

Example of adding a job:

import { addJob } from 'graphile-worker';

await addJob('task_name', { key: 'value' });

Recurrent Tasks

Graphile Worker supports scheduling recurrent tasks using a crontab file. To configure recurrent tasks:

1. Create a crontab file in the root directory.
2. Define tasks in the file using the cron syntax.

Example crontab file:

# Run the "cleanup" task every day at midnight
0 0 * * * cleanup

# Run the "send-emails" task every hour
0 * * * * send-emails

3. Ensure the crontab file is loaded by the worker when the application starts.

For more details, refer to the Graphile Worker Crontab Documentation.

API Routes

The application exposes the following routes under the /worker endpoint:

POST /worker/job/single

• Description: Adds a single job to the queue.
• Request Body: None.
• Response: Returns the job details after it is added to the queue.

POST /worker/job/batch

• Description: Adds a batch of jobs with the same identifier and different payloads to the queue.
• Request Body: None.
• Response: Returns the details of the batch jobs added to the queue.

POST /worker/job/bulk

• Description: Adds multiple jobs with different identifiers and payloads to the queue.
• Request Body: None.
• Response: Returns the details of the bulk jobs added to the queue.

Tasks

The application defines the following tasks in the tasks folder:

hello.task.ts

• Task Name: hello
• Description: Handles jobs with the hello identifier. It processes the payload and logs the details. If the payload is an array, it processes each item in the array.
• Code:

import { Injectable } from "@nestjs/common";
import type { Helpers } from "graphile-worker";
import { Task, TaskHandler } from "nestjs-graphile-worker";

@Injectable()
@Task("hello")
export class HelloTask {

	@TaskHandler()
	handler(payload: any | any[], _helpers: Helpers): void {
		if (Array.isArray(payload)) {
			_helpers.logger.debug(`handle array payload ${JSON.stringify(payload)}`);
		} else {
			_helpers.logger.debug(`handle ${JSON.stringify(payload)}`);
		}

		if (Math.random() < 0.5) {
			throw new Error("Random error occurred");
		}
	}
}

recurrent.task.ts

• Task Name: recurrent-task
• Description: Handles recurrent tasks with the recurrent-task identifier. It logs the payload details.
• Code:

import { Injectable } from "@nestjs/common";
import type { Helpers } from "graphile-worker";
import { Task, TaskHandler } from "nestjs-graphile-worker";

@Injectable()
@Task("recurrent-task")
export class RecurrentTask {

	@TaskHandler()
	handler(payload: any, _helpers: Helpers): void { 
		_helpers.logger.debug(`handle recurent task ${JSON.stringify(payload)}`);
	}
}

Resources

• NestJS Documentation
• Graphile Worker Documentation
• NestJS Discord Community

Acknowledgments

This project uses the nestjs-graphile-worker library for seamless integration with Graphile Worker.

Support

This project is open-source and licensed under the MIT License. Contributions are welcome!

For more details, see the LICENSE file.

Author

• Fernando Moro - GitHub Profile

---