fix: increase the default for proxy requests

[shaneutt]

What this PR does / why we need it:

Increases the default proxy timeout based on experiences in GKE (which was failing).

Which issue this PR fixes

Adds fixes that supports #1095
Supports the analysis found in #1519

PR Readiness Checklist:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[mflendrich]

Can we use some retry library instead of implementing retries ourselves? The second best thing to do is unit test the retry mechanism implemented in this PR but I think it would not be good use of our time.

[shaneutt]

Can we use some retry library instead of implementing retries ourselves? The second best thing to do is unit test the retry mechanism implemented in this PR but I think it would not be good use of our time.

Given that there's any contention I've opted to remove the retry mechanism from this PR in favor of focusing on the better default so we can get this fix quicker.

[codecov]

Codecov Report

Merging #1610 (6a3c65c) into next (f56d00e) will increase coverage by 0.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             next    #1610      +/-   ##
==========================================
+ Coverage   62.35%   62.56%   +0.20%     
==========================================
  Files          82       82              
  Lines        7180     7180              
==========================================
+ Hits         4477     4492      +15     
+ Misses       2372     2361      -11     
+ Partials      331      327       -4     

Flag	Coverage Δ
integration-test	62.56% <100.00%> (+0.20%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
internal/manager/config.go	92.50% <100.00%> (ø)
internal/proxy/clientgo_cached_proxy_resolver.go	82.11% <0.00%> (+1.32%)	⬆️
...trollers/configuration/zz_generated_controllers.go	34.63% <0.00%> (+2.03%)	⬆️
internal/ctrlutils/utils.go	95.83% <0.00%> (+2.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f56d00e...6a3c65c. Read the comment docs.

[ccfishk]

lgtm.