Celery Fixes Worker Stalling and Redis Memory Leaks

The Celery project recently addressed several critical stability issues affecting distributed task execution and backend resource management. Recent updates focus on preventing worker process stalling during broker reconnections and fixing a silent memory leak in the Redis result backend.

A significant fix in the async pool implementation resolves a condition where workers could stall or lose concurrency after a broker connection loss. Previously, the flush operation in the async pool would unconditionally clear the busy worker set. This forced the fair scheduler to lose track of which processes were still executing accepted tasks.

When the broker reconnected, the scheduler would see those busy workers as available and attempt to write new tasks to their pipes. This caused new tasks to queue behind long running jobs even when other workers in the pool were idle. The fix introduced by Arkadiusz Bach in commit d4eb32a changes this behavior to trim the busy set instead of clearing it. The pool now filters the busy worker list to retain only those file descriptors backed by still active processes. This preservation of the busy worker invariant ensures that the fair scheduler remains synchronized with the actual state of the prefork pool.

The Redis result backend received a fix for a memory leak that occurred when waiting for pending results. When the result consumer polls for a task that has already reached a ready state, the internal meta data was being buffered in a map that never evicted the entry. This happened because the result was already resolved and removed from the tracking set before the state change logic could process it.

The update in commit 74a7a63 introduces a shortcut for single result lookups. If a task is already finished, the consumer now cancels the underlying pub/sub subscription without attempting to buffer the message for a result that is already gone. This change is specifically scoped to single task results to ensure that collective result sets still follow the full state change path required for group logic.

Several improvements landed for how workers handle queue subscriptions and consumer gate logic. One notable fix addresses the prefetch gate behavior when using the worker disable prefetch setting. Upon connection loss, the consumer would clear its reserved requests set. This caused the gate logic to mistakenly report that the worker could accept more messages, even if a task from before the disconnect was still running.

The project corrected this in commit 1cc9ecf by ensuring that the active requests set is used to update the reserved set during the closing phase. This keeps the prefetch gate closed while the pool is genuinely busy. Additionally, commit 7f9a6e5 fixed an issue where adding back a cancelled queue would fail to restore the consume from state, ensuring that dynamic queue management through the remote control API is more reliable.

The task execution pipeline now has better resilience when running error handlers. A change in commit 477c816 wraps new style errback calls in a try except block. This prevents a single failing error handler from halting the entire chain of remaining handlers, matching the defensive behavior of the legacy errback implementation.

For users running tasks across timezone boundaries, the time utility was updated to handle Daylight Saving Time gaps. When a local time is made aware in a timezone during a spring forward transition, the time might not technically exist. The update in commit c30f42a uses the dateutil library to resolve these imaginary times to a valid local time, preventing value errors during task scheduling.

Operators running Celery with the Redis backend or using the worker disable prefetch option should look toward the next maintenance release. The fixes for worker stalling during broker cycles are particularly important for environments with long running tasks where fair scheduling is enabled.

If you use custom error handlers, verify that they do not rely on side effects from previous handlers in the chain, as the execution is now more isolated. The transition to codecov action version 7 in the CI pipeline indicates a move toward more modern tooling for the project infrastructure.