Add watch request timeout to prevent watch request hang

[xigang]

What type of PR is this?
/kind bug

What this PR does / why we need it:
When the federate-apiserver's watch request to the member cluster gets stuck, it will cause the watch request from the federated client to get stuck as well.

Which issue(s) this PR fixes:
Fixes #5672

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ikaven1024 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/search/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 57.14286% with 9 lines in your changes missing coverage. Please review.

Project coverage is 40.90%. Comparing base (331145f) to head (5e6e24b).

Files with missing lines	Patch %	Lines
pkg/search/proxy/store/multi_cluster_cache.go	57.14%	8 Missing and 1 partial ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5732   +/-   ##
=======================================
  Coverage   40.90%   40.90%           
=======================================
  Files         650      650           
  Lines       55182    55196   +14     
=======================================
+ Hits        22573    22579    +6     
- Misses      31171    31179    +8     
  Partials     1438     1438           

Flag	Coverage Δ
unittests	40.90% <57.14%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xigang]

cc @XiShanYongYe-Chang @RainbowMango @ikaven1024 PTAL.

[zhzhuang-zju]

@xigang Hi, if a watch request is hanging and causes a timeout, will the hanging watch request continue to exist in the subprocess?

[xigang]

@zhzhuang-zju Yes, there is this issue. When a watch request times out, the goroutine needs to be terminated.

[RainbowMango]

Good point! Then that case we have to cancel the context passed to cache.Watch().

[RainbowMango]

So this patch intends to terminate the hanging by raising an error after a period of time. Is this the idea?

[RainbowMango]

Another question:
Before starting the Watch, we tried to get the cache of that cluster, I'm curious why this cache still exists even after the cluster is gone. Do we have a chance to clean the cache?

karmada/pkg/search/proxy/store/multi_cluster_cache.go

Lines 333 to 336 in e7b6513

	cache := c.cacheForClusterResource(cluster, gvr)
	if cache == nil {
	continue
	}