feat(ipc) avoid new workers attempting to replay evicted events

[thibaultcha]

A follow-up to #88.

Described in #93 and reported a few months ago is the likely possibility that mlcache and ipc instances are created in the init phase. If a worker is to be started much later in the master process' lifetime, the newly forked ipc instance will have an idx attribute set to the shm_idx value at the time of init (likely 0). These workers will resume polling evicted events, and poll() will likely timeout indefinitely from there.

For the fix in #88 to work, the mlcache instance has to be instantiated during init_worker or later.

This patch proposes an approach which works for instances created in both init and init_worker: as soon as the ipc shm has started evicting items, we guarantee that future workers will resume polling at the current index, without having to call any method but poll().

Fix #93

[thibaultcha]

A better approach here would be if shm:incr() accepted flags, or maybe even an shm:set_flags() API. In the absence of a solution to rely on flags, we rely on an extraneous key, and on it never reaching the LRU tail.

[juancampa]

Wouldn't it be enough to run this on the first call to poll?

if self.idx < 0 then
  self.idx = shm_idx
end

🤔

[thibaultcha]

If we do this, we'd introduce a regression and we would prevent the first workers from polling any early broadcasted event, see for example this test. Since what we're worried about are missing (evicted) events which would cause streaks of timeouts, I should still be safe to preserve this behavior that if the event at index 1 is still present, then we can just replay them all fast enough.

[thibaultcha]

Does it still sound good to you @juancampa? Something you'd care testing maybe? I doubt it for such a minor change but just checking-in in case.

[juancampa]

What I hesitate a bit on is that the behavior is significantly different between workers that are spawned before the first eviction and workers that are started after an eviction. A simple example would be:

a worker that is started, after 10000 events but before the eviction happened. Wouldn't that worker go through all 10000 events? Compare that to a worker that was started after 10000 events but after an eviction. This one wouldn't catch up at all.

For mlcache purposes, do we need to replay old events at all. If a worker is new, it won't have the keys being deleted by the broadcast in L1 anyway.

[thibaultcha]

Well, like I said above, new/fresh workers need to run past events before the first poll() in the current implementation, and I did not want to break this atm. Keeping track of which workers are new or which ones are re-spawning could be done but I think this is already a worthy improvement without too complicated of a patch. Better have a worker replay past events even if it may timeout a few times along the way as it catches up, rather than never catch up at all.

If there isn't any other way, maybe we can change this behavior in a major release so that all ipcs first poll() always skip to the current index, for mlcache purposes only.