Skip to content

feat(redis)!: native async cluster Pub/Sub on redis-py 8 - #3131

Merged
IvanKirpichnikov merged 2 commits into
1.0.0from
feat/redis-cluster-native-async-pubsub
Sep 23, 2026
Merged

IvanKirpichnikov merged 2 commits into
1.0.0from
feat/redis-cluster-native-async-pubsub

Conversation

@Lancetnik

@Lancetnik Lancetnik commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Closes #3009. Targets 1.0.0 rather than main, because it raises the redis floor.

What this removes

Redis Cluster Pub/Sub has been running through a synchronous client since #2854, because redis.asyncio.cluster had no publish / pubsub at the time. redis-py 8.0.0 added them, so the workaround goes:

  • _SyncPubSubProxy, sync_publish(), _get_sync_cluster() and the ThreadPoolExecutor in configs/state.py — along with the _sync_cluster / _thread_pool fields that the base ConnectionState carried for the cluster's sake
  • RedisClusterFastProducer in publisher/producer.py, whose channel publish hopped to a thread on every message
  • _make_channel_subscriber() and the subscriber() override in broker/cluster_broker.py, which existed only to monkeypatch start() on channel subscribers

Cluster now uses the same RedisFastProducer and the stock ChannelSubscriber.start() as standalone. BaseRedisFastProducer collapsed into RedisFastProducer since there is no longer a second producer to share with.

The credential_provider gap @fbartonoctopus reported on #3009 closes as a side effect: the hardcoded option whitelist in _get_sync_cluster() is gone, so every connection option reaches Channels, Lists and Streams through the one async client.

The dependency decision

The issue left this open. This PR takes option 2 — raise the floor to redis>=8.0.0 — rather than the runtime guard #2893 used. One supported path, and it lets faststream/redis/_compat.py go entirely: REDIS_V710 and REDIS_V720 are now always true, so the claim_min_idle_time version guard, the lib_name/lib_version fallback in _get_driver_info(), and the require_redis_v710 test marker all disappear with it.

This is breaking for every Redis user, not just cluster users — hence 1.0.0.

One real bug the live cluster surfaced

ClusterPubSub.execute_command reads nodes_manager.slots_cache directly instead of going through RedisCluster.execute_command, so it never triggers the client's lazy topology discovery. A channel subscriber is usually the first cluster operation an app performs, and it died on KeyError: <slot>. RedisClusterConnectionState.connect() now awaits client.initialize().

That makes connect() reach the network, which is what "connect" should mean — and it caught four tests that were only passing because the old connect was a no-op:

  • test_start_stop_lifecycle ran RedisClusterBroker() against the default localhost:6379 with no cluster there; it is now connected-marked and points at the real cluster
  • test_pipeline_warns published without starting the broker, which worked only because the sync client built itself lazily from _options; it now starts the broker, matching what standalone RedisBroker already required
  • test_broker_gets_patched_attrs_within_cm and test_broker_with_real_doesnt_get_patched come from the shared BrokerTestclientTestcase and start the broker for real. The cluster overrides carry connected now, which moves them from test-redis-cluster-smoke, where no cluster runs, to test-redis-cluster-real

Checked during implementation

The issue flagged three things worth confirming, since #2893 predates current main:

  • legacy_responses=True — not needed. redis-py 8 already defaults RedisCluster(legacy_responses=True), so RESP3 still yields RESP2-shaped responses and stream/list parsing is untouched. Passing it explicitly would be noise.
  • refactor(redis): replace deprecated args: lib_name, lib_version to driver_in… #2956 (driver_info) — already landed; _get_driver_info() just loses its pre-7.2 branch.
  • Sharded Pub/Sub — out of scope, as the issue says.

Validation

  • ruff format / ruff check / codespell / mypy (1746 files) / pyright clean
  • tests/brokers/redis + tests/docs/redis, in-memory: pass
  • tests/brokers/redis -m redis against a live standalone Redis: pass
  • tests/brokers/redis/cluster -m redis_cluster against a live 3-node cluster (redis 8, ports 7001-7003): pass

@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file Redis Issues related to `faststream.redis` module and Redis features labels Sep 14, 2026
Comment thread faststream/redis/broker/cluster_broker.py Outdated
Comment thread faststream/redis/configs/state.py Outdated
@Lancetnik
Lancetnik force-pushed the feat/redis-cluster-native-async-pubsub branch from ad65ad2 to 2275ac7 Compare September 23, 2026 05:18
Lancetnik added a commit that referenced this pull request Sep 23, 2026
The five how-to pages send contributors to the navigation file by its
GitHub URL on `main`. On 1.0.0 that file moved to
`docs/docs/en/SUMMARY.md`, a path `main` gains only when this release
ships, so lychee answers 404 and takes the `docs` gate with it.

It went unnoticed because PR #3125 is a draft and every test job skips
drafts; #3131 is the first non-draft PR on this base to run the check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
redis-py 8.0.0 gave the async `RedisCluster` `publish` and `pubsub`, so the
synchronous workaround from #2854 can go: no second Redis client, no
`ThreadPoolExecutor` in an otherwise async path, and no cluster-only code
branch wherever Pub/Sub is involved.

`RedisClusterFastProducer`, `_SyncPubSubProxy`, `sync_publish()`,
`_get_sync_cluster()` and the `_sync_cluster` / `_thread_pool` fields on the
base `ConnectionState` are gone. Cluster now uses the same `RedisFastProducer`
and the stock `ChannelSubscriber.start()` as standalone, which also means every
connection option — `credential_provider` for IAM / token auth included —
reaches Channels, Lists and Streams through one client.

BREAKING CHANGE: `faststream[redis]` now requires `redis>=8.0.0`.

Closes #3009

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Lancetnik
Lancetnik force-pushed the feat/redis-cluster-native-async-pubsub branch from 2275ac7 to 16c4143 Compare September 23, 2026 05:25
…tch auto-connect

`_get_driver_info()` only existed to branch between `driver_info` and
`lib_name`/`lib_version`; with the `redis>=8` floor it is a constant.

`RedisClusterBroker.publish_batch` was the last publish path that connected
implicitly — cluster `publish` and standalone `publish_batch` both require a
started broker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IvanKirpichnikov
IvanKirpichnikov merged commit 60db560 into 1.0.0 Sep 23, 2026
30 checks passed
@IvanKirpichnikov
IvanKirpichnikov deleted the feat/redis-cluster-native-async-pubsub branch September 23, 2026 17:37
Lancetnik added a commit that referenced this pull request Sep 23, 2026
The instance-level `start` replacement goes away with the native async cluster
Pub/Sub (#3131, on the 1.0.0 branch); the class and its `exclude-classes` entry
say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lancetnik added a commit that referenced this pull request Sep 23, 2026
Conflicts, resolved to 1.0.0's version:
- `faststream/_internal/fastapi/*`: the FastAPI plugin is gone on 1.0.0; main only
  added `__slots__` there.
- `faststream/redis/configs/state.py`, `faststream/redis/publisher/producer.py`:
  1.0.0 dropped the sync cluster proxy and the separate cluster producer (#3131);
  main's `__slots__` (#3220) are kept on the classes that remain.
- `pyproject.toml`: 1.0.0's `requires-python` and version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation Redis Issues related to `faststream.redis` module and Redis features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants