HTTP health check endpoint

[bouk]

Hi @bensheldon, thanks for the great gem.

We're using good_job in an environment that requires an HTTP health check to make sure the service is healthy. For Sidekiq there's a plugin available that does something similar: https://github.com/arturictus/sidekiq_alive

I couldn't figure out a good way to build a plugin like this for good_job, so I was wondering if you've dealt with this before and have any tips, or could suggest a good way to integrate an endpoint like this with good_job.

[bensheldon]

I am really excited by this challenge. Both because I think it's operationally important to have probes, and because it's an interesting technical challenge.

For the webserver part of the probe:

• If you're using GoodJob in async mode (i.e. running jobs inside the webserver), you can add the code below to your webserver's existing probe.
• For the CLI, it would need to have a tiny server built into it. That seems like it would be 20 lines of code to do with Rack... though I'm curious what the tiniest Rails API server implementation would be because the CLI is already initializing Rails (probably YAGNI).

For the content of the probes:

• Readiness: the job schedulers have initialized and are ready to work jobs:

  GoodJob::Scheduler.instances.all?(&:running?)

• Liveliness: I have experienced Ops people arguing about what a Liveness probe should entail and whether it should check on service dependencies (e.g. database). This sort of thing. If I accept that liveliness includes a working database connection, then I would use:

  GoodJob::Scheduler.instances.all?(&:running?) &&
    GoodJob::Notifier.instances.all?(&:listening?)

@bouk Is this a project you'd want to do? I think the smallest scope would be adding a simple Rack server (like this), starting the Rack server inside of a Concurrent::Future thread somewhere around here, and maybe making it conditional on some configuration like good_job.config.probe_port.

[bouk]

Maybe the readiness check should check that it can reach the database (so the old worker can be shut down) and the liveliness should check less, since a failure will lead to the worker being killed.

Thanks for the pointer on how to check the health, I can make an attempt.

[bensheldon]

Thanks! This makes me think that, given my fear that the definition of readiness and liveliness will be debatable, that the probe endpoints should be descriptive of GoodJob. e.g. /status/running (which is descriptive of whether GoodJob has initialized) and /status/listening (which is descriptive of GoodJob having an active connection to the database) and that way the developer can choose how to configure their probe.

[aried3r]

This sounds like a good feature to have! I just wanted to chime in to say the currently proposed checks would also return true for the empty array. We are in the process of upgrading our app from Rails 6.0 to 6.1 and notices that with 6.1 the instances do not get started, but that's a different story and why I'm excited about this feature. :)

irb(main):001:0> GoodJob::Scheduler.instances.all?(&:running?)
=> true
irb(main):002:0> GoodJob::Scheduler.instances
=> []

irb(main):005:0> GoodJob::Notifier.instances.all?(&:listening?)
=> true
irb(main):006:0> GoodJob::Notifier.instances
=> []

Edit: Just to be clear, our Rails 6.1 problem is not related to this gem but our way of loading and initializing.

[bensheldon]

@aried3r Thanks! Yes, it should be GoodJob::Scheduler.instances.any? && GoodJob::Scheduler.instances.all?(&:running?)

Can you open an issue for what you're seeing about the instances not being started?

[bensheldon]

Linking a Reddit thread of another developer seeking similar functionality: https://reddit.com/r/ruby/comments/q4bbd3/ruby_webserver_started_in_background_thread/

[bensheldon]

@bouk I released this healthcheck feature in v2.7.0:

https://github.com/bensheldon/good_job#cli-http-health-check-probes