xdsclient: fix TestServerFailureMetrics_BeforeResponseRecv test to wait for watch to start before stopping the listener

[purnesh42H]

Fixes: #8211

TestServerFailureMetrics_BeforeResponseRecv was flaking because of race between listener watch to start and closing the listener and if watch hasn't started the stream errors won't be received and server failure metric won't be emitted. Adding the hook to wait for stream to open before stopping the listener have fixed the race.

RELEASE NOTES: None

Successful forge run

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.08%. Comparing base (5edab9e) to head (9d2ba7f).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8217      +/-   ##
==========================================
- Coverage   82.17%   82.08%   -0.10%     
==========================================
  Files         410      410              
  Lines       40236    40236              
==========================================
- Hits        33065    33028      -37     
- Misses       5822     5852      +30     
- Partials     1349     1356       +7     

see 22 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arjan-bal]

This code is still racy because the backoff could have completed before the listener is restarted. There is an option that can be used to set a really long backoff duration instead.

grpc-go/xds/internal/xdsclient/pool.go

Lines 60 to 63 in 5edab9e

	// StreamBackoffAfterFailure is the backoff function used to determine the
	// backoff duration after stream failures.
	// If unspecified, uses the default value used in non-test code.
	StreamBackoffAfterFailure func(int) time.Duration

[purnesh42H]

So, in this particular test case it won't happen because we are setting the watch expiry timeout which is way shorter than the back off. So, watch will just expire before backoff anyway but before that we would received the stream error.

[dfawley]

This does ~nothing now - delete it?

And before, it should have been written as: if ctx.Err() != nil {...}

[purnesh42H]

yeah not needed because both lis.Stop() and lis.Restart() are serialized by lock.

Replaced the similar select block in the other test with if ctx.Err() != nil {...}