Skip to content

Don't log caller cancellation as a distributed lock error (#608) - #626

Open
qjustfeelitp wants to merge 1 commit into
ZiggyCreatures:mainfrom
qjustfeelitp:fix/issue-608-distributed-lock-caller-cancellation
Open

Don't log caller cancellation as a distributed lock error (#608)#626
qjustfeelitp wants to merge 1 commit into
ZiggyCreatures:mainfrom
qjustfeelitp:fix/issue-608-distributed-lock-caller-cancellation

Conversation

@qjustfeelitp

Copy link
Copy Markdown

Summary

Fixes #608.

When the caller's own CancellationToken (e.g. HttpContext.RequestAborted) is canceled while the distributed lock accessor is waiting to acquire the lock, Medallion.Threading surfaces an OperationCanceledException / TaskCanceledException driven by that token. FusionCache caught it in the generic catch (Exception) handler of DistributedLockerAccessor.AcquireLock/AcquireLockAsync and logged it as:

[DL] acquiring the DISTRIBUTED LOCK has thrown an exception
System.Threading.Tasks.TaskCanceledException: A task was canceled.

at the configured DistributedLockerErrorsLogLevel (Error by default) — even though nothing actually went wrong with the distributed locker. It was simply the caller walking away.

Root cause

Medallion.Threading's BusyWaitHelper distinguishes the two outcomes by design (DistributedLock.Core/Internal/BusyWaitHelper.cs):

  • acquire timeout → returns null (the lock could not be taken within timeout);
  • caller cancellation → throws OperationCanceledException driven by the caller's token.

So an OperationCanceledException whose cause is the caller's own token is not a locker failure — it is normal caller cancellation. The factory path in FusionCache already treats caller cancellation this way (it rethrows the OperationCanceledException before reaching the generic catch). The distributed lock accessor was the one place that didn't, so the same logical event was logged as an error on one path and handled cleanly on the other.

Fix

Add the same guard the factory path uses to both the sync and async distributed lock accessors: when the exception is an OperationCanceledException and the caller's token is the one that was canceled, rethrow it so the cancellation flows to the caller instead of being logged as a distributed locker error. A genuine locker fault (or a cancellation not tied to the caller's token) still falls through to the existing catch (Exception) handler and is logged/handled exactly as before.

catch (OperationCanceledException) when (token.IsCancellationRequested)
{
    // CALLER CANCELLATION: the caller's own token was canceled (eg: HttpContext.RequestAborted)
    // while waiting to acquire the distributed lock. This is not a distributed locker error, so
    // it must not be logged as one: just let the cancellation flow to the caller, consistently
    // with how caller cancellation is handled on the factory path.
    throw;
}

Tests

Adds DistributedLockerTests with a sync and an async test. Each installs a fake IFusionCacheDistributedLocker that reproduces the Medallion behaviour — it cancels the caller's token while "waiting" and then throws an OperationCanceledException driven by that token — and asserts that:

  1. the cancellation surfaces to the caller as an OperationCanceledException, and
  2. the [DL] acquiring the DISTRIBUTED LOCK has thrown an exception message is not logged.

Verified as genuine regression tests (A/B against this branch):

[DL] error logged? Test result
without the fix yes (the reported symptom) ❌ 2 failed
with the fix no ✅ 2 passed

Notes on #608

The original report attributed the flood of errors to a shared CacheKeyPrefix across instances. That is a red herring (as already noted in the thread): the errors are simply many concurrent requests whose callers get canceled while parked on the distributed lock during heavy contention (in the reporter's case, 50+ app instances against one shared cache under a UI-test suite). This change removes the misleading error logging at its source without touching keys or prefixes.

…ures#608)

When the caller's own CancellationToken (eg: HttpContext.RequestAborted)
is canceled while the distributed lock accessor is waiting to acquire the
lock, Medallion.Threading surfaces an OperationCanceledException driven by
that token. FusionCache caught it in the generic exception handler and
logged it as "[DL] acquiring the DISTRIBUTED LOCK has thrown an exception"
at the configured error level, even though nothing actually went wrong
with the distributed locker.

The factory path already rethrows caller cancellation before the generic
catch; the distributed lock accessor was the only place that didn't. Add
the same guard to the sync and async accessors so caller cancellation
flows to the caller instead of being logged as a locker error. A genuine
locker fault still falls through to the existing handler unchanged.

Adds sync + async regression tests using a fake locker that reproduces the
Medallion behaviour (cancels the caller token, then throws OCE driven by
it): they assert the cancellation surfaces to the caller and the "[DL] ...
has thrown an exception" message is not logged.

Fixes ZiggyCreatures#608
@jodydonetti jodydonetti self-assigned this Jul 23, 2026
@jodydonetti jodydonetti added the bug Something isn't working label Jul 23, 2026
@jodydonetti jodydonetti added this to the v2.6.1 milestone Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Distributed locks timeout

2 participants