More intelligently detect if a plugin should be killed, rather than waiting for X seconds

[Thunderforge]

Sometimes, a Gauge plugin takes a long time to run. Gauge makes an effort to not freeze the user's machine and so attempts to preemptively kill a frozen plugin.

Expected behavior

Gauge checks if a plugin is still doing work, such as by examining the time stamp of log files. If there has been work in the last few seconds, assume that the plugin is still active.

Actual behavior

Gauge forcibly kills a plugin after the number of seconds determined by the plugin_kill_timeout property (by default: 4 seconds), regardless of if the plugin is actively doing work. The code for this begins on this line in plugin/plugin.go.

This has especially caused issues with the HTML Report plugin on large projects (see HTML Report #120, #121, #153, as well as this project's #636). Theoretically, it could happen with any plugin, but so far, no users have reported that other plugins have run for longer than the default 4 seconds.

Gauge version

Gauge version: 1.0.0
Commit Hash: 5a99965

Plugins
-------
csharp (0.10.3)
html-report (4.0.5)
java (0.6.8)
json-report (0.2.1)
screenshot (0.0.1)
xml-report (0.2.0)

[sriv]

Thanks for bringing this up, @Thunderforge

Here is a proposal:

• The plugins send a "keep-alive" ping back to gauge every N seconds, (N can be X-1).
• Gauge will reset the timer when it receives a "keep-alive" ping.
• This should result in gauge killing the plugin only if it does not hear anything for over the plugin_kill_timeout interval.

Thoughts?

[Thunderforge]

@sriv That sounds like an excellent way to do it. Obviously, it would mean updating all the plugins to send a "keep-alive" ping, but I think that in the long run, this is a better solution.

[sriv]

tech notes

There are (at least?) two possible ways to implement a "keep-alive"

Gauge core orchestrated

1. gauge sends a "should I keep you alive" request ?
2. plugin responds with a "yes, please!"
3. gauge resets the timer
4. not done? - goto (1)
5. the plugin is killed when it fails to respond to "should I keep you alive".

plugin initiated

1. plugins knows it will be killed in plugin_kill_timeout time interval.
2. plugin sends a "wait I am not done yet" request to gauge just before the timeout is about to expire.
3. gauge resets the timeout timer.
4. not done? - goto (2)

/cc @getgauge/core - please add your thoughts.

If we do the Gauge core initiated then plugins need to have additional metadata (ex. capabilities) that should be honoured by Gauge before sending these requests.

[nehashri]

I prefer the first approach where Gauge core sends a should I keep you alive request to the plugins. This seems more in terms of our current communication approach where the communication happens from Gauge to the plugins and plugins only respond to requests by Gauge.
But this also means that the plugins need to keep listening to the requests sent by Gauge. A single threaded approach may not work here as plugin may not be able to respond to Gauge immediately.