…gether (#3270)

The sequence number was taken with Interlocked.Increment and the timestamp read
separately afterwards, so two attempts completing at once - the interactive and
subscription connections failing together - could interleave: one takes the
earlier sequence, the other takes the later sequence but reads the clock first,
and a later SequenceNumber carries an earlier CompletedTimeUtc. Handlers that
order outcomes by sequence (as ConnectionAttemptUnitTests.EveryFailedReconnectIsReported
does) then see time going backwards.

Both are now taken under one lock. The event is rare - once per physical
connection attempt - so the lock costs nothing that matters. Found on the v4
branch, where the two connections are dialled concurrently and the test failed
about 2 runs in 5.

…3272)

The overload (and StringSetAsync via inheritdoc) pointed at the DELEX
command page; it was copied from StringDelete. Also add the missing
<returns> to match the sibling overloads.

Fixes #3269

* Tolerate refused CLUSTER SLOTS/NODES during topology updates (#3274)

Proxied deployments (Azure Managed Redis / Redis Enterprise) can identify
as a cluster while refusing CLUSTER SLOTS ("ERR command is not allowed";
Enterprise exposes SLOTS only under the OSS cluster API). That refusal
surfaced as a logged *error* from GetEndpointsFromClusterNodes at connect
and on every reconfigure, including the periodic topology refresh.

A refusal is a posture, not a fault: catch RedisServerException per task -
SLOTS refused falls back to the NODES-only path (as the method's own
comment always intended), NODES refused means no topology view at all, so
return null; both are logged at Information (EventIds 128/129).

Tests: ClusterCommandsRefusedUnitTests drives the in-process test server
with SLOTS (and SLOTS+NODES) refused, asserting discovery still works and
no error is logged; CapturingLogger promoted from ConnectFailureRefreshTests
to a shared test helper.

* Observe both topology tasks, so an abandoned one never faults unobserved

SLOTS and NODES are issued as a pair; if one await throws anything other
than a server refusal, the other task is abandoned mid-flight - and both
ride the same connection, so the orphan is likely to fault too. Attach
ObserveErrors at creation so it can never surface as an
UnobservedTaskException in the host. (Pre-existing gap: the bare
sequential awaits had the same shape.)

… is reported (#3273)

* Investigate Redis Enterprise lag-aware availability checks

Confirms the premise: this is the client-side geographic failover feature
family, and Lettuce, Jedis and redis-py have all shipped it. It feeds exactly
the health checks our Availability/ namespace already has behind SER007.

The abstractions line up almost one-to-one with redis-py's - probe, policy
(all/any/majority), circuit breaker, per-member config - and two details make it
land cleanly: HealthCheckProbe has no internal abstract members, so it is
already subclassable from outside the assembly, and ConnectionGroupMember
already carries a per-member HealthCheck override, which is what a per-cluster
REST endpoint needs.

Where we are better placed: HealthCheckResult has Inconclusive, which Lettuce's
two-state HealthStatus lacks. Lettuce maps a failed REST call to UNHEALTHY, so a
management-plane outage is indistinguishable from the data plane being down and
can fail over a database that is serving fine. We can say "could not
determine" instead.

The awkward part is that src/ has no HTTP client and no JSON parser. JSON turns
out to be avoidable if the bdb uid is configured rather than discovered, since
the availability call is a status-code check.

Also records the defaults disagreeing across clients: 100ms everywhere -
server, docs, redis-py - but 5000ms in Lettuce.

* Record what a Tunnel API for the management plane should look like

mgravell offered to add one; the shape is what decides whether it helps.

A member returning HttpMessageHandler would drag System.Net.Http into
StackExchange.Redis's public API for the sake of one commercial deployment
target - which is the thing keeping this in a separate package was meant to
avoid, decided the expensive way. A stream-shaped member instead composes into
SocketsHttpHandler.ConnectCallback, which takes exactly a ValueTask<Stream>, so
the Enterprise side gets a complete HTTP hijack and core gains no HTTP types at
all. Tunnel already speaks that language via BeforeAuthenticateAsync and
ConnectTransportAsync, and adding a virtual is binary-safe for subclasses.

Two functional arguments for Tunnel over a separate knob, beyond testing: a
CONNECT proxy configured for Redis almost certainly applies to the Enterprise
REST API too, so co-locating makes that correct by construction; and the test
harness already assigns Tunnel, so one assignment would spoof both planes.

Caveats recorded: ConnectCallback is net5+, the existing members are
EndPoint/ConnectionType-shaped where HTTP wants a Uri, and widening Tunnel's
remit means HttpProxyTunnel and LoggingTunnel each need an opinion about whether
they apply to the management plane.

Also retires open question 5: RawConfig and ConfigurationOptions.Tunnel are both
already public, so a probe reaches the tunnel from the context it already gets -
no HealthCheckContext change needed.

* Real clusters and the fault injector, and a correction

I had written that inducing "reachable but stale" on demand is not something
anyone can do in CI, so the fakes were the only route to the scenario that
matters. That is wrong. Redis ships a fault injection service that the client
suites drive over HTTP - POST /action with a type and parameters, poll the
returned action_id - and Lettuce's ActiveActiveFailoverScenarioTest injects
network_latency against a bdb_id for a bounded duration, which is exactly the
mechanism for pushing replication lag past the tolerance and watching the
lag-aware check flip while the plain check stays green.

So the authoritative end-to-end test is real cluster plus fault injector, on a
road Lettuce has already built. The fakes are demoted rather than removed: they
still cover machines with no Enterprise cluster, they are far faster than an
injector action that budgets minutes plus a stabilisation wait, and they cover
what the injector does not obviously reach - specific error_code bodies, 401/403
credential expiry, slow-but-successful responses for probe timeouts.

Also records what the real clusters are uniquely good for: everything in section
1 is read off documentation, and a short session settles whether extend_check=lag
is actually supported, what failure bodies really contain, whether the Host
header is required, what the bdbs listing really returns - which decides whether
the JSON-avoidance argument holds - and what lag real geo links show, which is
the empirical way to settle 100ms versus 5000ms.

* We already have a fault-injector tier; it answers several of the open questions

It is not merged - tests/StackExchange.Redis.FaultInjector.Tests lives on
marc/maint-optin-server, open as #3191 (107 files, +13160) - which is presumably
why I did not find it when I grepped a branch taken fresh from main.

Two pieces of it bear directly on this. FaultInjectorClient wraps the same
POST /action + GET /action/{id} contract go-redis, redis-py and node-redis
converged on, and adds GetValidTriggersAsync, so we can ask a deployment which
lag-inducing effects it supports instead of guessing at an action name. And
ClusterRestClient is already a Redis Enterprise REST client on 9443 - Basic
auth, System.Text.Json, GET /v1/bdbs?fields=uid,name - so the availability call
is a method on an existing class, not a new project.

It also settles by observation what I had listed as open and sourced only from
documentation: the management certificate is self-signed per environment and
pinned to a CA from the config directory, so the probe needs CA configuration
and cannot rely on the ambient trust store; auth really is Basic; and the bdbs
listing returns the requested fields, so discovery is easy - which weakens the
avoid-JSON argument to "it keeps JSON out of src/", not "discovery is hard".

Sequencing consequence recorded: an end-to-end test is downstream of #3191.
And #3191's own summary is the best argument for validating before building -
same vendor, same class of API: the specifications are prose, the payloads were
never published, and nearly every starting assumption was wrong.

* Add the plan, blocked on #3191

Step 0 is to wait for #3191 to merge and build from there, rather than writing a
second Redis Enterprise REST client against the same cluster and reconciling the
two later - its ClusterRestClient already does Basic auth, CA pinning and
GET /v1/bdbs, and its FaultInjectorClient can be asked which lag-inducing
effects a deployment supports. The one thing that can start earlier is the
API validation, because it produces notes rather than code.

Step 1 is that validation, before any implementation: a checklist of things
currently taken from documentation, including the measurement that settles
100ms versus 5000ms, with captured bodies saved as fixtures so the stubbed tests
replay real payloads.

Step 2 deliberately does not decide where the code lives, but records that the
dependency argument is weaker than it looked - System.Net.Http needs no package
reference on any target we ship - so the question is judgement, not constraint.

Also records that this branch becomes the feature branch when step 0 clears, and
that the way to take the merge is to rebase onto main rather than merge main in,
since #3191 squash-merges.

* Step 0 is cleared; note that step 1 needs the machine with cluster access

#3191 merged as dc915bb, so this branch is rebased onto it - which is what step
0 asked for, and rebased rather than merged because #3191 squash-merged and a
merge would have moved the merge base without bringing its ancestry. The
fault-injector tier is present here now.

Also recorded where the work can happen. Step 1 is validation against a real
deployment plus the fault injector, and both live on the machine holding the
environment directory; this box has no ~/aws and no env_output.json anywhere, so
nothing in step 1 could be run or verified here. Worth writing down rather than
rediscovering, since the environment being absent looks identical to the
environment being offline. Steps 2 onward are ordinary code and are not tied to
a machine.

Noted too that the directory is the AWS multi-cluster template, so env_output.json
nests its outputs under .clusters.value[0] - a shape FaultInjectorEnvironment
already handles, so it needs no work, just no surprise.

* Pin the cluster REST certificate per host rather than chaining to the config CA

The 9443 certificate is self-signed by the cluster itself; the CA in the config
directory only signs the RESP proxy certificates, so the chain build could never
pass. Pin the leaf on first use, keyed by host so multi-cluster templates work.

* Step 1 measured: the lag check is Active-Active only, and does what it says there

Records the real-deployment results (findings section 10): off Active-Active the
lag check fails unconditionally; on it, the stale member flips within seconds of
sync pausing and the tolerance is honoured on a running link. Also the two
integration problems found in Availability/, which mean src/ needs a change.

* Correct the client defaults: Lettuce, Jedis and redis-py all default to 5000 ms

Read from source: redis-py's 5000 contradicts its own docstring's 100. None
of the three checks for Active-Active, and none has a third result state.

* Measure what a probe can discover: not Active-Active over RESP, db_viewer suffices

Active-Active membership has no RESP signal, so it comes from REST (fields=crdt)
or configuration. db_viewer can call everything a probe needs, but unfiltered
bdb/crdb reads expose database passwords even to it, so only filtered requests.
The REST endpoint derives from the configured database hostname.

* TLS reuse, the cross-client survey, and the local form routed by database host

TrustIssuer(leaf) via TlsOptions validates the 9443 call unchanged. No other
client activates lag checks automatically or checks crdt; go-redis routes the
local form through each database host, which makes it truthful, and scopes the
lag verdict to failback only.

* The local form flips under paused sync exactly as the database form does

Same verdict, same timing, stale member only, tolerance ignored while paused.
Its failure code is bdb_endpoint_unavailable, shared with the wrong-node case,
so the choice between forms is about call shape, not information.

* Sketch the lag-aware API for review

A failback-only health check per member rather than a new result value, a
member-scoped probe, and a LagAware factory whose only required input is the
management credential. Compared against Lettuce, Jedis, redis-py and go-redis.

* API sketch: System.Text.Json decided; open questions compared with other clients

Keep LagAware naming and failback (not fallback) terminology; rotating
NetworkCredential callback plus a separate client-certificate option; and keep
the liveness fallback, which no client has, because a dead peer probably makes
the survivor's lag check fail.

* Partition test: the lag check is blind for a minute, then fails on both sides

Partitioning the clusters (tagged iptables rules, self-healing) showed no lag for
~60 s while sync had stopped, then tolerance-governed failure on both members;
a tolerance of 0 is honoured. Supports the liveness fallback, and adds a caveat:
failback in the first minute after a link breaks is not protected.

* AllSuccess: a run of only inconclusive probes is not a pass

The default policy reported Healthy when no probe failed, even if none
succeeded. Harmless for liveness, where both count as connected, but wrong for
any caller that distinguishes them.

* Failback health checks: ask whether a member has caught up before moving to it (SER011)

ConnectionGroupMember.FailbackHealthCheck (and a group default) is consulted
only for members that are not active, and only Healthy makes them eligible. It
never evicts the active member, and when no live member is eligible, selection
falls back to liveness, since a partition leaves every region behind at once.
HealthCheckProbe.Scope lets a probe run once per member rather than per endpoint.

* Notes: core change implemented behind SER011 (SER007 is retired)

* HealthCheck.LagAware: ask Redis Enterprise whether a member is available and caught up (SER011)

A member-scoped probe of the database availability API. Only the management
credentials are required: the REST endpoint derives from the configured host,
the database is discovered with filtered requests only, the lag question is
asked only of Active-Active databases, and no tolerance is sent unless set.
Could-not-ask is Inconclusive, never Unhealthy. System.Text.Json on the
down-level targets; custom certificate validation is unavailable on net461.

* Notes: probe implemented; RawConfig is not public after all

* LagAware: ask the cluster name, and never follow redirects

Found against live clusters: non-master nodes answer the management routes
(discovery, the crdt lookup) with a 307 to the master's internal address, so
deriving the REST endpoint from the database host timed out for three of four
members. Derive the cluster FQDN (strip redis-{port}.), which resolves to the
master, and report a redirect as Inconclusive rather than following it.

* Notes: management routes redirect to the master's internal address

* Fault-injector scenario: refuse to fail back onto a stale Active-Active member

Pauses sync into the preferred member through its cluster's REST API while
writes go to the other, and asserts the group stays off it for 15s although it
is alive, then fails back once it has caught up. The tier now reads every
cluster's credentials, and shares its certificate pin with the probe.

* Notes: tier-3 scenario test passing

* LagAware: client certificates for the REST call

LagAwareOptions.ClientCertificate, with SetUserPfxCertificate and (net6.0+)
SetUserPemCertificate mirroring ConfigurationOptions and sharing its loaders.
Wired into SocketsHttpHandler on .NET and HttpClientHandler down-level; not
supported on net461. Every other client offers mTLS for this call.

* Failback: a member chosen by the liveness fallback is provisional

Other clients never select a member that fails its lag check; the nearest
equivalent that still serves when every member is behind is to leave such a
member for an eligible one as soon as one is available, whatever the weights,
and to keep checking it so it becomes an ordinary active member once caught up.

* Notes: fallback choice is provisional, as decided by client parity

* Move the lag-aware log message to a free event id

main now uses 124 and 125 (#3257); the guard test from #3266 rejects duplicates.

* AnySuccess: a run of only inconclusive probes is not a failure

AnySuccess reported Unhealthy whenever nothing succeeded, even when nothing
failed either - so a probe that cannot tell (a key-write probe on a replica,
say) marked its member unhealthy. Now Inconclusive, matching AllSuccess and
MajoritySuccess.

* Move the lag-aware log message to a free event id (again)

main has since taken 128 as well.

* Failback: a zero FailbackDelay clears a failure in the same clock tick; no ValueTuple

UpdateState compared the last failure strictly before the cutoff. With
FailbackDelay zero the cutoff is now, and on .NET Framework's ~15.6ms clock a
failure and the passing check that clears it often share a tick, holding the
member out for another interval. Compare inclusively. Also replace a named tuple
in the failback-check pass: the library must not reference System.ValueTuple.

* Ship lag-aware failback without an experiment

Remove SER011: the attributes, the constant and NoWarn entry, the [SER011] API
prefixes and docs/exp/SER011.md. It never shipped, so it is not a reserved id.
Also restores main's using directives and encoding in the files that already
existed there, which earlier edits had disturbed.

* Notes: no experiment

* Notes: drop the SER011 history