Better scheduler configuration

[ThomasCrambert]

Hello 👋,

Currently, there’s no configuration option to specify if a scheduler should be started or not, this is decided if a recurring task file is given or not.

In our case, we would like to have only one scheduler for better isolation. To do that, we have to use the following configuration:

production:
  workers:
    - processes: 0
      threads: 0

This will explicitly tell solid_queue to not boot anything apart from the supervisor and since we also give it a recurring task file, it will also start a scheduler.

This workaround works fine for us but I feel like there could be something more intuitive. If you think the same, I can try to come up with a solution. 👍

[rosa]

Hey @ThomasCrambert, this is almost wired up but not exactly what you want, via these not yet documented options.

If you don't specify any workers, it also works, you don't need to set processes and threads to 0:

In our case, we run the dispatchers and scheduler separately from workers with this configuration:

# config/solid_queue/dispatching.yml

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
    - polling_interval: 2
      batch_size: 500

We have recurring.yml in the default place. Then we just run this:

bin/jobs -c config/solid_queue/dispatching.yml

And that starts the 2 dispatchers and 1 scheduler. If you want only 1 scheduler, then perhaps we need an only_recurring option (besides the empty configuration workaround).

[ThomasCrambert]

Hello @rosa 👋,

Thanks for your quick reply 🙏

you don't need to set processes and threads to 0:

Setting processes and threads to 0 is to avoid any default value:

• processes will ensure no additional process will be started aside from the supervisor and the scheduler, otherwise an additional worker and dispatcher would be started 😕
• threads is to avoid this check to fail (the estimation will give 5 since no value would specified in the configuration, I was planning on opening a separated issue for this)

only_recurring option (besides the empty configuration workaround).

That would be awesome 🙏

[rosa]

Ah, I meant that you can leave the workers attribute out completely, so you wouldn't need to specify these values (it'd behave in the same way as processes: 0 and threads: 0), just like in the configuration example I pasted above.

[rosa]

This is now better documented in the README, with the addition of a new environment variable to completely skip the scheduler as well, added in #580.

[rosa]

Oh wait, I got this the other way around. You wanted to have just the scheduled, instead of no scheduler 😅 Going to reopen for that --only-scheduler option.