[leo_storage][leo_mq] Decreased a performance of read and write operation during rebalance/recover-node

[yosukehara]

We took some benchmarks for LeoFS v1.2.10 and then we found performance degradation as follows:

• benchmark report (4th June, 2015)

[windkit]

I think it would be better if we could sort out two sets of parameters. (Under reference system configurations)
One for aggressive recovery, that is utilizing all resources (or most, e.g. 80%)
One for conservative recovery, that is utilizing minimal resources (e.g. <20%)

Administrators could then choose which one they want, either full-speed recovery during maintenance or auto-triggered recovery without affecting foreground tasks much.

On top of it, dynamic configuration would be great to have.

[yosukehara]

@windkit Thank you for your comment. From last night, I've been checking MQ's consumption mechanism because it seems MQ is not able to handle mq-consumption's interval parameter correctly.

[mocchira]

@windkit +1.
From administrator's point of view,
The current configurations related to MQ|Watchdog might be difficult to set appropriate values corresponding to their own environment.
So wrapping those difficult configurations by what you suggest would make sense and
needless to say, dynamic configuration great.

After @yosukehara finish to fix this issue,
We will discuss how configurations should be exposed for admins.

[yosukehara]

Regarding the dynamic configuration, we only implement the watchdog mechanism, as you mentioned, we need to provide the mq and other mechanism similar to the watchdog.

[yosukehara]

We've taken a benchmark of recover-node w/front operations as below: