Redis caching backend deadlocks on dispose when notification thread never started (unbounded wait)

[gfraiteur]

Summary

PostSharp.Patterns.Caching.Backends.Redis can deadlock on dispose (unbounded, no timeout) when the Redis pub/sub notification-processing thread was never started. A transient Redis connect/subscribe hiccup is enough to trigger it. The stuck thread is the (single) finalizer thread, which then prevents the host process from exiting.

This is the product root cause behind the CI test-target hang reported in #22 (the TimeSensitiveTest target runs the Caching/Redis tests, and a hung finalizer keeps the test host alive until the build is killed). Same source code passes or hangs depending only on the runtime Redis condition — i.e. it is not a code regression, it is a latent unbounded-wait bug.

Affected component

Patterns/Caching/PostSharp.Patterns.Caching.Backends.Redis/RedisNotificationQueue.cs
(also the sync-over-async dispose in RedisCachingBackend.cs)

Root cause

In RedisNotificationQueue.InitAsync the channel-subscription loop runs before the processing thread is started:

• RedisNotificationQueue.cs:86-114 — SubscribeAsync + while (!IsConnected(channel)) (with a connectionTimeout) + PingAsync, all of which can throw on a transient hiccup or cancellation.
• RedisNotificationQueue.cs:116 — notificationProcessingThread.Start(...) only runs after the loop succeeds.

notificationProcessingThreadCompleted (a TaskCompletionSource) is signalled only in the processing thread's finally block (RedisNotificationQueue.cs:254-262). If the subscription loop throws, the thread never starts, so that TCS is never completed.

Any subsequent dispose then blocks forever:

• RedisNotificationQueue.cs:352 (sync Dispose) — notificationProcessingThreadCompleted.Task.Wait() — no timeout, no cancellation.
• RedisNotificationQueue.cs:353 / :426 — notificationProcessingThread.Join() — unbounded.

Because a failed RedisCachingBackend.Create / CreateAsync does not dispose the partially-initialized backend (RedisCachingBackend.cs:240, :280), the orphaned RedisNotificationQueue is collected and its finalizer (~RedisNotificationQueue -> Dispose(false)) hits the unbounded wait on the finalizer thread. A permanently blocked finalizer thread stops all finalization and prevents clean process exit.

Impact

• A transient Redis pub/sub connect failure can hang the disposing/finalizing thread indefinitely.
• In a host that depends on clean shutdown (e.g. a test host, or a short-lived process), this manifests as a process that never exits.
• Customer-facing risk: applications that create Redis caching backends under flaky network conditions can leak a wedged finalizer thread.

Fix direction

1. Always complete notificationProcessingThreadCompleted even when the processing thread never starts (e.g. set it on the init-failure path, or only wait when the thread was actually started).
2. Bound every dispose wait (Task.Wait/Thread.Join) with a timeout; log and proceed on timeout rather than blocking forever.
3. Dispose the partially-initialized backend/queue when Init/InitAsync throws, so failures clean up deterministically instead of relying on the finalizer.
4. Add a deterministic regression test (init fails before thread start -> dispose must return promptly).

Relation

• Root cause of / surfaces as PostSharp.Patterns TimeSensitiveTest target deadlocks on startup (reproducible hang in threading tests, regression on 2026-06-10) #22 (PostSharp.Patterns TimeSensitiveTest target deadlocks on startup). PostSharp.Patterns TimeSensitiveTest target deadlocks on startup (reproducible hang in threading tests, regression on 2026-06-10) #22 is the CI symptom; this issue is the underlying product defect.

[gfraiteur]

Proven fix exists in Metalama — back-port candidate.

The same code lineage in Metalama (Metalama.Patterns.Caching.Backends.Redis.RedisNotificationQueueProcessor) already fixed this class of deadlock in commit 0174948329 "Fixed RedisNotificationQueueProcessor." (2024-07-04). Metalama is not affected because its dispose:

• guards the completion-wait with if ( _notificationProcessingThread.IsAlive ) — so when a connect/subscribe failure means the processing thread never started, the wait is skipped entirely (no hang);
• bounds the wait to 5 s with a warning before any unbounded fallback;
• adds a null sentinel to the BlockingCollection to wake the blocking Take(), and drops the unbounded Thread.Join().

PostSharp's RedisNotificationQueue.Dispose/DisposeAsync lack all of these guards. Back-porting the Metalama pattern is the recommended fix.

(Metalama issue tracker metalama/Metalama has no dedicated issue for this — the fix landed directly in code. Cross-codebase porting is routine, cf. metalama/Metalama#1164 "port changes from PostSharp".)

[gfraiteur]

Fix in progress: postsharp/PostSharp#43 (into release/2026.0). Ports Metalama's IsAlive-guarded dispose, adds a deterministic regression test, and adds a per-test-project vstest watchdog timeout as defense-in-depth. Verified locally against a real Redis (repro hung 15s pre-fix; passes post-fix; existing 37 RedisCacheBackendTests still pass).

[gfraiteur]

Production impact

This is not test-only — the same defect can hit customer applications.

Who is exposed: any app using the Redis caching backend with events/invalidation enabled (SupportsEvents = true), which is required for cross-node cache invalidation — the usual reason to use Redis caching in a multi-instance app. A plain non-invalidating Redis cache never creates the notification queue and is not affected via this path.

Trigger: a transient pub/sub subscription failure during backend creation — RedisNotificationQueue.InitAsync throws in the SubscribeAsync / while (!IsConnected(channel)) (→ TimeoutException) / PingAsync block before notificationProcessingThread.Start(). Realistic causes: Redis briefly unreachable or mid-failover at app startup; network/DNS/TLS latency exceeding the subscription connection timeout; AbortOnConnectFail = false (common) where Connect returns and the interactive PingAsync succeeds but the subscription is not ready, so the IsConnected loop runs out the clock.

Mechanism:

1. InitAsync throws before the thread starts, so notificationProcessingThreadCompleted is never signalled.
2. The exception propagates out of RedisCachingBackend.InitAsync before this.notificationQueue is assigned, so the backend's own Dispose skips it and the calling thread just receives the exception (it does not hang).
3. The half-initialized RedisNotificationQueue is orphaned and unreferenced → GC finalizes it → ~RedisNotificationQueue → Dispose(false) → notificationProcessingThreadCompleted.Task.Wait() blocks forever, on the process's single finalizer thread.

Impact of a wedged finalizer thread:

• All finalization stops process-wide → finalizable objects pile up and are never collected → steadily growing memory and unmanaged-handle leaks (eventual OOM / handle exhaustion).
• Graceful shutdown hangs (teardown waits on finalizers): in containers the orchestrator graceful-stop times out and SIGKILLs; under IIS, w3wp hangs on recycle. This is the same "host never exits" failure seen on CI.
• Silent and cumulative: the app may catch/retry the init exception and appear healthy; retried attempts orphan more queues that also never finalize.

What it does not do: it does not directly hang a request/foreground thread — the calling thread gets the exception; only the finalizer wedges. So the production symptom profile is "creeping leak + hung shutdown," not "requests hang."

The CI test-suite hang and this production failure are the same defect; the fix (wait for the notification thread only when it was actually started) closes both.

[gfraiteur]

Load-testing the dispose fix under CPU starvation

The original CI hang reproduced only intermittently, so before finalizing the fix we stress-tested the dispose path directly under forced thread preemption.

Because the deadlock lived entirely in the RedisNotificationQueue dispose handshake — and the notification-processing thread itself never touches Redis (it blocks on a BlockingCollection and only invokes a delegate) — we were able to drive the real thread-alive dispose path with no live Redis: constructing the queue with an empty connection/channel set starts a genuine processing thread while skipping all Redis I/O.

Harness:

• 6,400 disposes (400 rounds × 16 concurrent), alternating the synchronous Dispose() and asynchronous DisposeAsync() paths;
• run under a CPU burner oversubscribing all 24 cores at 200% (48 busy loops), to force preemption at arbitrary points in the CompleteAdding → thread-completes → WhenAny/Wait → Join sequence.

Result: all 6,400 disposes returned promptly — no deadlock. The contention stretched the run from ~1s to 4m37s (~270×), confirming the threads were heavily preempted throughout.

This load-test scaffolding was reverted after verification. It required temporarily widening an internal initialization method purely for the test and spun up thousands of short-lived OS threads, so it was not appropriate to retain in the committed regression suite. The committed regression test (RedisNotificationQueueDisposeTests) covers the deterministic root-cause path — disposal when the notification thread was never started — while the load test served only to give additional confidence in the thread-alive path.

[gfraiteur]

Released in v2026.0.12.

— Claude