Introduction

How does rethinkdb-job-queue work?

To understand how the job queue works you first need to understand the components. There are two JavaScript objects used to manage the queue and process jobs. These objects are the Queue and Job objects.

What is a Queue object?

The main component of rethinkdb-job-queue is the Queue object.

Throughout this documentation you will see references to either the "Queue object" or the "queue". They are not the same thing. So what do these terms mean?

• "Queue object" is referencing the local JavaScript object created by calling;
• const q = new Queue().
• "queue" is referencing the list of jobs that need to be processed. It is the named table in the database.

As you can see above, it is easy to create a Queue object. Here is a more complete example.

const Queue = require('rethinkdb-job-queue')
const cxnOptions = {
  host: 'localhost',
  port: 28015,
  db: 'JobQueue'
}
const qOptions = {
  name: 'EmailJobs'
}
const q = new Queue(cxnOptions, qOptions)

The above example is connecting to a RethinkDB database named JobQueue hosted on the localhost at port 28015. The name of the Queue object is being set to 'EmailJobs' by changing the Queue Options. This code is also connecting to the queue of jobs in a table named 'EmailJobs'.

If this is the first time you have created a Queue object called 'EmailJobs' then the following will take place as part of the instantiation of the Queue object.

• Does the database exist? No, create it.
• Does the table in the database called 'EmailJobs' exist? No, create it.

What we just created is a queue by using a Queue object.

Alternatively, if the queue already exists the following will take place.

• Does the database exist? Yes.
• Does the 'EmailJobs' table in the database exist? Yes.

In this case, the new Queue() instantiation call is not creating the queue but connecting to it.

With this in mind, if the documentation talks about the Queue object it is talking about a single JavaScript object. If the documentation talks about the queue it is talking about the table in the database hosted by RethinkDB.

There can be many Queue objects hosted either within one Node.js process, across multiple Node.js processes in the same computer, or across multiple Node.js processes in multiple computers.

What is a job?

Right. So we know what a queue is now. That's a good start. Let's shine some light on jobs.

Within the queue we create jobs. A job can either be data in the backing table waiting to be processed, or a JavaScript Job object within your code populated with the data from the database. It would be very unusual to interact with the job data in the database so think of a job as a Job object in your code.

For a job to be processed with a rethinkdb-job-queue Queue object, it needs to have a minimum set of information. Here is what a newly created job looks like in the database.

Using this code:

const Queue = require('rethinkdb-job-queue')
const q = new Queue()
let job = q.createJob()
q.addJob(job).catch(err => console.error(err))

We get this job in the database:

{
  "dateCreated": Sat Aug 20 2016 21:26:32 GMT+00:00 ,
  "dateEnable": Sat Aug 20 2016 21:26:32 GMT+00:00 ,
  "id":  "2250b4fc-5167-49ae-a139-eff48b7b1580" ,
  "log": [{
    "data": null,
    "date": Sat Aug 20 2016 21:27:03 GMT+00:00 ,
    "message":  "Job added to the queue" ,
    "processCount": 0
    "queueId":  "WebDev:rjqJobQueue:rjqJobList:18532:93ea5586-f840-4b03-9acc-ef3fc9260858" ,
    "retryCount": 0 ,
    "status":  "waiting" ,
    "type":  "information"
  }] ,
  "priority": 40 ,
  "processCount": 0,
  "progress": 0 ,
  "queueId":  "WebDev:rjqJobQueue:rjqJobList:18532:93ea5586-f840-4b03-9acc-ef3fc9260858" ,
  "repeat": false ,
  "repeatDelay": 300000 ,
  "retryCount": 0 ,
  "retryDelay": 600000 ,
  "retryMax": 3 ,
  "status":  "waiting" ,
  "timeout": 300000
}

By looking at the basic schema of a job above it will give you an understanding of what a job is. There is a lot of information here and it is explained in more detail elsewhere within the documentation. With that said there are a couple of points worth looking at.

• The job and the processing Queue object can be easily identified.
• There are a number of date values for chronological job management.
• Job processing can be controlled with the job priority.
• There are a number of values to manage job failure and retry.
• Processing history is recorded.
• Job status is easy to see.

If you closely analyze the job data above you should notice something missing. This job is basically useless. There is nothing in the job data related to the actual job you want to process. No email addresses, no processing numbers, no file paths.

One of the interesting features of rethinkdb-job-queue is the ability for you to decorate your JavaScript Job objects with any properties you want or need for job processing. The properties you place on your Job objects will be stored in the database for later processing by a Queue object. Here is an example storing email details.

job.recipient = 'batman@superheros.com'
job.subject = 'New Gadget'
job.body = 'Your new fancy gadgets are ready to be picked up.'