Respond to watches in parallel

[snowp]

Currently java-control-plane uses an EmitterProcessor to notify subscribers that a watch has changed. Since this executes the the subscription callbacks synchronously, we end up spending a substantial time constructing the gRPC responses while holding the SimpleCache write lock during setSnapshot, essentially making the cache inaccessible until all the watches have been triggered.

It would be nice to use an async processor instead (maybe WorkQueueProcessor?) which would allow us to decouple the response construction from watch triggers, reducing overall lock contention.

Spent a bit of time figuring this out but ran into some deadlocks and issues in test, so filing this for now to track the the issue in case anyone else runs into this.

[joeyb]

Yeah, I’ve spent some time trying to refactor that code, but haven’t got it in a state I’m happy with yet. We aren’t really doing much with reactor, so ideally I’d like to drop that dependency entirely.

[snowp]

It'd be pretty cool to get rid of reactor. Considering we're not even making use of their parallel processing features it's not really adding much value.

Happy to help in any way I can, as the current blocking nature of watch triggers are a bit problematic when a lot of watches are triggered at once, so we're trying to avoid any larger issues with this as we scale out our deployment.

[lvijay]

@snowp, @joeyb can this be closed given that #65 dropped reactor?

[joeyb]

Fixed by #85.