App retries

[azzaea]

A Swif/t App may fail nondeterministically, and be tried to run in the same host and MPI rank using the TURBINE_APP_RETRIES directive.
It may be useful however to attempt to run the app in a different rank (in case one of the hosts is unavailable, or there is network issue or the like). This pull request: attempts to run the app (upon failure) in a different MPI rank.

A simple test is added, where an external app (an infinite loop ) is run in the background, and turbine attempts to kill it twice. This works fine the first time, but when attempting to kill it again, it either fails (if no retries are allowed), or retries TURBINE_APP_RETRIES times (in different ranks) and quits upon failure.

Another simple test creates a file, then attempts to delete it twice. Creation and deletion both work in the first time, and the second deletion attempt would retry on different MPI ranks until reaching TURBINE_APP_RETRIES times , at which point the script exits.

[azzaea]

Upon a closer look, an app is delayed either:

• before it is launched by setting, TURBINE_APP_DELAY ( new feature ). This is via the app_delay function by round($app_delay_time * 1000.0 * rand())

• between retries when it fails the first time and until it either succeeds or TURBINE_APP_RETRIES are reached. In this recent commit This is achieved via the app_delay_retries, by $app_backoff * pow(2, $tries) * rand()

I think this behavior is sensible in both cases. Any ideas or comments?

[j-woz]

I would like to merge this. Can you add some user control to enable or disable this feature, say by an environment variable TURBINE_APP_REPUT ? Re-put means the app call is retried with another ADLB_Put. Other name suggestions or controls are welcome. I think this feature should be disabled by default because it would be unexpected, but well documented and easy to enable.

[j-woz]

Also note we do brace-after-newline in C.
http://swift-lang.github.io/swift-t/dev.html#_c_code

[azzaea]

You are right, will follow the convention more closely.

If I'm following correctly, the behavior could be:
TURBINE_APP_RETRIES=n ==> A failing app is retried on the same rank for n times (old behaviour)
TURBINE_APP_REPUT==> If this was set to 1, then the n retries are put to different ranks.

However, maybe it is easier (from a user perspective) to set the retry behavior via single parameter.. As in:
TURBINE_APP_RETRY_LOCAL=n ==> A failing app is retried on the same rank for n times (old behavior)
TURBINE_APP_RETRY_REPUT=n ==> A failing app is retried on other ranks (besides its own) for n times (this pull request)

Would you have a preference among the 2?

[j-woz]

I like your second idea- it is neatly categorized.

[azzaea]

Great! The latest pull request would now retry an app either locally or on a new rank, depending on which TURBINE_APP_RETRY_LOCAL=n or TURBINE_APP_RETRY_REPUT=n is set.

If both directives were specified, only the local retries will be done. Not sure if the user should get a warning (in the log) in this case?

[azzaea]

Pull request merged here, so closing this issue