How to avoid SPoF?

[SWSAmor]

Is there anyone use queue in production? If yes please help me how to do that!
One queue instance is a single point of failure. So we need 3 or more. But with Tarantool master-master replication maybe they won't be in sync - based on what i read about. Inconsistent tubes sounds very bad to me.
I read about vshard and Avitos autovshard but even if it's good for failover than I have to fork this repo and replace all of the DMLs to the vshard.router.call. ?
I'm very uncertain about that. I appreciate any guidance.

[Totktonada]

Sorry for the long delay.

The way to go is to setup replica that will do just box.cfg(<...>) and fetch data. When the old master is going to be a replica, it should be restarted to stop processing tasks. When the old replica is promoted to master it should wait for all data from the past master (tarantool/tarantool#3808) and call require('queue') to start processing tasks on this instance.

I know, this looks weird. I hope we should implement #60 instead.

[LeonidVas]

IMHO, this only be done with qsync(AFAIU this is Alexander's proposal), but in this case you will degrade the performance of the "queue". It seems like this implementation of "queue" is not designed for this purpose.

[Totktonada]

IMHO, this only be done with qsync(AFAIU this is Alexander's proposal), but in this case you will degrade the performance of the "queue". It seems like this implementation of "queue" is not designed for this purpose.

Nope. And, as you correctly stated, it would lead to significant performance degradation (at least in latency).

A project, where I participate in the past, uses queues for different purposes and each microservice has a replica. In fact, we don't even have automated wait_lsn() and doing it manually. An instance is always started either as replica (with bare minimum of Lua code) and as a master (with full application) and becomes in this state until restart.

It was okay for us, because even with manual switchover we have enough availability to be under required SLA.

The project lands to production with tarantool-1.7.6.47 first time. There was no synchronous replication in tarantool at the moment.

[SWSAmor]

Thank you the answers.
It's a financial institution so the manual switchover isn't enough. Async also not. We mustn't lost any transaction.
We have service mesh with Consul. Two sites and every service is equal and duplicated per site except queue. Queue is the SPoF. That is what need to be solved.
By the way the queue is working flawlessly for months so that is rather theoretic, auditing or SLA problem. But everything can fail. :-)

[Mons]

Exceptionally durable queue may be prepared with synchronous replication and quorum failover (raft). But that's requires at least 2.8.2.
@Totktonada, possible an example of such config should be attached

[Totktonada]

@Mons I would start from the manual switchover (without instance restart). I think about this a bit and, it seems, there are quite large amount of tricky details. Even without full automation.

---

Let's consider scope of this issue as defined below.

• Provide a way to opt out of immediate tasks processing after require('queue'), so it may be called on a replica. Or even always require explicit queue.start().
• Provide a way to gracefully stop processing of tasks: queue.stop().

It should be enough to switch master-replica manually when there is no parallel processing of tasks on two or more instances ('master-replica' case¹) — without restart. Let's consider #60 as a task about 'true master-master'.

The queue.stop() should forbid to take tasks. TBD: Think re (un)burrying, delaying and other operations. I guess everything except ack and release should be forbidden, but may be there are valid use cases I don't aware of.

I think that queue.stop() should give a change for taken tasks to be acked (wait till some timeout) and forcibly return the tasks into the 'ready' state if the timeout expired. I think we can lean on task's ttl / ttr and global ttr here. We can also provide a timeout option: queue.stop({timeout = <...>}): it is possible that someone, who didn't bother about timeouts before, will stuck on queue.stop() at the moment when (s)he needs to switch master and replica.

Take care to the following case: someone calls queue.stop(), found that it stucks, run another console and calls queue.stop({timeout = 1}). Both calls should return after one second. More generally: all currently executed queue.stop(<...>) calls should return, when one of them (with minimal timeout) returns.

Or... a client that uses queue's session mechanism may ack a task that was taken on a previous master. Should we expect from a client that it'll reconnect to the new master (possibly after attempt to ack on the past master) and ack the task there? If so, we should allow to distinguish 'I'm not a leader' error from all others and should not autorelease tasks. Don't know, need to think here. Maybe we should support both flows. If we'll support both, whether to choose 'whether to autorelease' automatically based on knowledge about authenicated and non-authenticated task owners or add an option to queue.stop() to release all tasks before return, including ones from authenticated owners? Or both?

The manual switchover scenario will be the following:

• Call queue.stop() on master and wait until it'll return.
• Wait until all data will land to a replica (all replicas): monitor box.info.vclock and box.info.replication.
• Demote the old master to replica: box.cfg{read_only = true}.
• Promote the old replica (one of replicas) to master: box.cfg{read_only = false}.
• Call queue.start() on the new master.

If we'll change the queue module to don't start tasks processing automatically after require('queue'), it will be incompatible change (all current users should add explicit queue.start() call into its code). I would think, whether we can keep backward compatibility here.

Should queue.start() wait until full box loading if it is called before first box.cfg()? Should it wait until the instance will become writeable if it is the read only state now? I guess that both answers should be 'no', because it looks error-prone.

---

Questions to think around:

• Can we automate the switchover using the RAFT leader election algorithm for asynchronously replicated queues?
• How about synchronously replicated queues with RAFT?

I have no clear vision, how it should be implemented to ensure correct automatic switchover.

Footnotes

1. Technically the replication may be configured as bidirectional, but there is a property (usually box.info.ro), which defines a leader, and only one leader at max may exists in a replicaset. ↩

[Totktonada]

However, maybe, just automatic that leans on ro/rw status would be even simpler and will cover most of cases. I need to think here.

[0x501D]

Proposal:

Create internal queue states:
[S]tartup | [R]eady | [E]nding | [W]aiting

• S:
  (state changed to RW)
  wait for maximum upstream lag * 2 (quiet for clients)
  start driver (extend drivers api required).
• R:
  run as usual
• E:
  (state changed to RO)
  stop driver.
• W:
  do nothing (monitor RO/RW status)

other notes:
implement per-task ttl.