scatter_gather

A Ruby gem that provides a scatter-gather pattern for coordinating ActiveJob execution. Jobs can wait for other jobs to complete before executing, with configurable polling, retry, and timeout behavior.

Tip

This gem was created for Cora, your personal e-mail assistant. Send them some love for allowing me to share it.

Usage

Start some scatter jobs and create a gather job that waits for all dependencies to complete:

class EmailProcessorJob < ActiveJob::Base
  include ScatterGather

  def perform(email_id)
    # Process email
  end
end

class AttachmentProcessorJob < ActiveJob::Base
  include ScatterGather

  def perform(email_id)
    # Process attachments
  end
end

class AICategorizerJob < ActiveJob::Base
  include ScatterGather

  def perform(email_id)
    # Categorize email with AI
  end
end

class NotifyCompleteJob < ActiveJob::Base
  include ScatterGather

  def perform(email_id)
    # Notify that all processing is complete
  end
end

# Start the scatter jobs
email_parser_job = EmailProcessorJob.perform_later(email_id: 123)
attachment_processor_job = AttachmentProcessorJob.perform_later(email_id: 123)
ai_categorizer_job = AICategorizerJob.perform_later(email_id: 123)

# Create a gather job that waits for all dependencies to complete
NotifyCompleteJob.gather(email_parser_job, attachment_processor_job, ai_categorizer_job).perform_later(email_id: 123)

The gather job will:

• Check if all dependencies are complete
• If complete: enqueue the target job immediately
• If not complete: poll every 2 seconds (configurable), re-enqueuing itself
• After 10 attempts (configurable): discard with error reporting

Configuration Options

• max_attempts: Number of polling attempts before giving up (default: 10)
• poll_interval: Time between polling attempts (default: 2.seconds)

# Example with custom configuration
TouchingJob.gather(jobs, poll_interval: 0.2.seconds, max_attempts: 4).perform_later(final_path)

Installation

Add the gem to the application's Gemfile, and then generate and run the migration:

$ bundle add scatter_gather
$ bundle install
$ bin/rails g scatter_gather:install
$ bin/rails db:migrate

Development

After checking out the repo, run bundle to install dependencies. The development process from there on is like any other gem.

License

This gem is made available under the MIT license

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/julik/scatter_gather.