Introduce tony.application.x.untracked.timeout to solve partial jobs hang

[zuston]

No description provided.

[oliverhu]

just tony.x.timeout? tracked tasks also need a timeout?

[zuston]

Refer #603

Why
On TFRuntime, I found two problems on tensorflow training on our production env.

Sometimes, when using tf estimator api, evaluator maybe wait the newest global step but chief has finished. So evaluator will hang. TF doc
Sometimes, worker will hang due to tf bugs, but chief has finished its task. So we need to kill workers which not finished.
Besides do we need to make job failed when some worker or evaluator not finished?

I think it can depends on user with config.

I think it's enough to solve the above problems by introducing untracked task timeout.

Do you have other ideas about possible problem?

[zuston]

Gentle ping @oliverhu

[oliverhu]

I think the tony.x.timeout notion is more generic and easier to comprehend. People can specify tony.evaluator.timeout=100s etc.

[oliverhu]

We discussed offline to have some generic task grouping and specify dependencies..
tony.groupA.timeout.aftergroupB = 10h etc.

[zuston]

Close it. Solved by #621