Implement injecting sleep in go failpoints

[serathius]

What would you like to be added?

Linearization tests only support injecting panic https://github.com/etcd-io/etcd/blob/main/tests/linearizability/failpoints.go#L33-L53, it would be useful to extend it to also support injecting a sleep. It should already be supported by gofail.

Main change that needs to be implemented is detecting that failpoint was executed. For panic it's simple, we wait for process to exit. Injecting sleep is much more subtle. I think this can be achieved by listing the failpoints via http endpoint.

Why is this needed?

This will allow for easier detection of race conditions in etcd code.

[ramil600]

I started working on this, and then noticed that gofail doesn't disable the fail points properly through the http DELETE request:
https://github.com/etcd-io/gofail/issues/26
PR submitted and linked to the same issue.

[serathius]

Thanks for looking into this!

[ramil600]

@serathius, sure no problem, currently I can register sleep fail point and check whether it was successfully enabled. However I am not sure when it should be deactivated. Once sleep is enabled in the code, it will trigger sleep every time when it is executed. PR: #14796

[serathius]

Wrote on #14796 that we care about failpoint execution not failpoint enablement.

[ramil600]

@serathius, one suggestion I have is resetting sleep duration value to 0 and deleting description in the terms field of the failpoint, once sleep is executed.
Then when you check it through http endpoint, it still exists(http status = 200), but it won't execute sleep anymore, unless randomly activated again. Please let me know if it is ok with you and I will send the PR to modify gofail library.

[lavacat]

@ramil600 Are you still working on this issue? I think this is going to be very useful.

[pchan]

I can take this up and can build on ramil600 PR : #14796

[lavacat]

@pchan assigned to you. To fix this you'll need etcd-io/gofail#37 first

[lavacat]

@ramil600 apologies for reassigning without checking with you first.
I think you've made a good progress on etcd-io/gofail#37. Hope we can merge it soon and unblock this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.