Add internal_query_count_limit configuration

[garyhtou]

This allows the internal count limit in Solid Queue to be configured using an internal_query_count_limit option.

MissionControl::Jobs.internal_query_count_limit = 5_000
# or
config.mission_control.jobs.internal_query_count_limit = 5_000

The default value is 500,000.

[garyhtou]

At the moment, only the Solid Queue adapter implements this configuration. Are there any plans to implement an internal count limit for the Resque adapter too?

If not, maybe this internal_query_count_limit config should be renamed to solid_queue.internal_query_count_limit. Thoughts?

[rosa]

At the moment, only the Solid Queue adapter implements this configuration. Are there any plans to implement an internal count limit for the Resque adapter too?

Not really, because Resque doesn't have a problem with counting jobs like Solid Queue has, thanks to the llen operation in Redis, that has O(1) time complexity. Unfortunately, this is not true for some DB storage engines, notably InnoDB 😅 This is why we had to introduce it for Solid Queue.

If not, maybe this internal_query_count_limit config should be renamed to solid_queue.internal_query_count_limit. Thoughts?

Yes, I think that could work, but I'd prefer to delay (or even avoid completely) having per-adapter configuration options if possible. Having the option doesn't mean the adapter needs to use it, so I think it's ok to leave it like this, without modifying Resque.

[garyhtou]

I'd prefer to delay (or even avoid completely) having per-adapter configuration options if possible.

I agree!

Having the option doesn't mean the adapter needs to use it, so I think it's ok to leave it like this, without modifying Resque.

Yeah, that makes sense 👍

[rosa]

Awesome, thank you so much 🙏