Skip to content

Add Redis Cluster broker support - #2854

Merged
Lancetnik merged 3 commits into
mainfrom
feature/redis-cluster-broker
May 25, 2026
Merged

Add Redis Cluster broker support#2854
Lancetnik merged 3 commits into
mainfrom
feature/redis-cluster-broker

Conversation

@powersemmi

@powersemmi powersemmi commented May 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR introduces RedisClusterBroker — a new broker for Redis Cluster, inheriting from RedisBroker with the same constructor API for drop-in compatibility.

Key features:

  • Redis Cluster connection via startup_nodes with auto-discovery
  • Standard Pub/Sub (PUBLISH/SUBSCRIBE, sharded=False) for single-node delivery
  • List and Stream operations with automatic hash-slot routing via RedisCluster client
  • XAUTOCLAIM support for cyclic re-consumption of pending stream messages (min_idle_time)
  • Full RedisBroker API compatibility — migrate by changing the class name
  • Pipeline warning — raises RuntimeWarning if pipeline= is passed (not supported in cluster)

Documentation:

  • New docs/docs/en/redis/cluster.md — feature comparison, migration guide, limitations
  • Cross-references from Pub/Sub, Streams, Pipeline, and Claiming pages

Fixes #1170

Type of change

  • New feature (a non-breaking change that adds functionality)
  • This change requires a documentation update

Checklist

  • My code adheres to the style guidelines of this project (just lint shows no errors)
  • I have conducted a self-review of my own code
  • I have made the necessary changes to the documentation
  • My changes do not generate any new warnings
  • I have added tests to validate the effectiveness of my fix or the functionality of my new feature
  • Both new and existing unit tests pass successfully on my local environment by running just test-coverage
  • I have ensured that static analysis tests are passing by running just static-analysis
  • I have included code examples to illustrate the modifications

@powersemmi
powersemmi requested a review from Lancetnik as a code owner May 4, 2026 15:41
@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file Redis Issues related to `faststream.redis` module and Redis features labels May 4, 2026
@powersemmi
powersemmi requested a review from s3ths1 May 4, 2026 15:41
@powersemmi powersemmi self-assigned this May 4, 2026
@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch 2 times, most recently from 3528945 to fc42629 Compare May 4, 2026 16:36
@github-actions github-actions Bot added the github_actions Pull requests that update GitHub Actions code label May 4, 2026
@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch 2 times, most recently from 1b035b0 to 2a2b32a Compare May 4, 2026 16:44
@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch 13 times, most recently from 298004c to cf6ae92 Compare May 11, 2026 09:51
@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch 4 times, most recently from ef6f1ca to 22fdfcc Compare May 14, 2026 13:17
@fbartonoctopus

fbartonoctopus commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown

I had an agent attempt this in this PR but I am told by @Lancetnik that this PR should be getting merged soon

Linking the code in case it is useful, and looking forward to seeing this merged 🙌

@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch from 22fdfcc to 77abc58 Compare May 24, 2026 04:24
powersemmi added 3 commits May 24, 2026 07:25
@powersemmi
Add Redis Cluster broker support
3f0fa91
@powersemmi
Refactor Redis connection state and consolidate type definitions
54333e9 
Extract `ClusterConnectionState` logic into the base `ConnectionState`
class
as a generic, and move shared broker type annotations into a new
`faststream/redis/schemas/types.py` module to reduce import clutter and
duplication.
@powersemmi
Address Redis Cluster PR review feedback
3a7778b 
- Unify TestRedisBroker and TestRedisClusterBroker: cluster bootstrap
  logic moved into TestRedisBroker._create_ctx, gated by an isinstance
  check on RedisClusterConnectionState.
- Use Sequence instead of list for RedisClusterParams.startup_nodes and
  the matching _resolve_url_options signature; mark RedisClusterParams
  total=False so the field stays optional.
- Mark cluster tests slower than 1s with @pytest.mark.slow (xautoclaim
  circular scanning, pubsub_more module, middleware classes, and the
  publish tests that wait on events).
@powersemmi
powersemmi force-pushed the feature/redis-cluster-broker branch from 77abc58 to 3a7778b Compare May 24, 2026 04:25
@powersemmi
powersemmi enabled auto-merge May 24, 2026 06:22
@powersemmi
powersemmi added this pull request to the merge queue May 25, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 25, 2026
@Lancetnik
Lancetnik added this pull request to the merge queue May 25, 2026
@Lancetnik
Lancetnik removed this pull request from the merge queue due to a manual request May 25, 2026
@Lancetnik
Lancetnik added this pull request to the merge queue May 25, 2026
Merged via the queue into main with commit c80a75b May 25, 2026
31 checks passed
@Lancetnik
Lancetnik deleted the feature/redis-cluster-broker branch May 25, 2026 20:40
@fbartonoctopus

fbartonoctopus commented May 26, 2026

Copy link
Copy Markdown

@Lancetnik Great to see this merged. Will you be tagging a release that includes this change?

@fbartonoctopus fbartonoctopus mentioned this pull request May 29, 2026
Closed
PAzter1101 added a commit to PAzter1101/faststream that referenced this pull request Jun 3, 2026
@PAzter1101
refactor(redis): move Sentinel support into a dedicated RedisSentinel…
1a7226b 
…Broker

Per review feedback on ag2ai#2895, Sentinel support now lives in its own
RedisSentinelBroker / RedisSentinelRouter instead of extra parameters on the
generic RedisBroker — mirroring RedisClusterBroker (ag2ai#2854), so each topology is
a distinct broker class and Sentinel params no longer leak into RedisBroker /
RedisClusterBroker.

- RedisSentinelConnectionState builds the client via Sentinel.master_for();
  the generic RedisBroker / RedisRouter keep their original API.
- RedisSentinelParams holds sentinels / sentinel_master_name / sentinel_kwargs.
- RedisSentinelRouter adds FastAPI Sentinel support (which RedisClusterBroker
  does not provide).
- Docs: docs/en/redis/sentinel.md + navigation entry.

Refs ag2ai#1812
@PAzter1101 PAzter1101 mentioned this pull request Jun 3, 2026
Merged
@fbartonoctopus fbartonoctopus mentioned this pull request Jun 8, 2026
Closed
@Lancetnik Lancetnik mentioned this pull request Jul 14, 2026
Merged
7 of 9 tasks
@Lancetnik Lancetnik mentioned this pull request Aug 10, 2026
Open
@Lancetnik Lancetnik mentioned this pull request Sep 14, 2026
Merged
Lancetnik added a commit that referenced this pull request Sep 23, 2026
@Lancetnik @claude
feat(redis)!: native async cluster Pub/Sub on redis-py 8
2275ac7 
redis-py 8.0.0 gave the async `RedisCluster` `publish` and `pubsub`, so the
synchronous workaround from #2854 can go: no second Redis client, no
`ThreadPoolExecutor` in an otherwise async path, and no cluster-only code
branch wherever Pub/Sub is involved.

`RedisClusterFastProducer`, `_SyncPubSubProxy`, `sync_publish()`,
`_get_sync_cluster()` and the `_sync_cluster` / `_thread_pool` fields on the
base `ConnectionState` are gone. Cluster now uses the same `RedisFastProducer`
and the stock `ChannelSubscriber.start()` as standalone, which also means every
connection option — `credential_provider` for IAM / token auth included —
reaches Channels, Lists and Streams through one client.

BREAKING CHANGE: `faststream[redis]` now requires `redis>=8.0.0`.

Closes #3009

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lancetnik added a commit that referenced this pull request Sep 23, 2026
@Lancetnik @claude
feat(redis)!: native async cluster Pub/Sub on redis-py 8
16c4143 
redis-py 8.0.0 gave the async `RedisCluster` `publish` and `pubsub`, so the
synchronous workaround from #2854 can go: no second Redis client, no
`ThreadPoolExecutor` in an otherwise async path, and no cluster-only code
branch wherever Pub/Sub is involved.

`RedisClusterFastProducer`, `_SyncPubSubProxy`, `sync_publish()`,
`_get_sync_cluster()` and the `_sync_cluster` / `_thread_pool` fields on the
base `ConnectionState` are gone. Cluster now uses the same `RedisFastProducer`
and the stock `ChannelSubscriber.start()` as standalone, which also means every
connection option — `credential_provider` for IAM / token auth included —
reaches Channels, Lists and Streams through one client.

BREAKING CHANGE: `faststream[redis]` now requires `redis>=8.0.0`.

Closes #3009

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lancetnik Lancetnik Lancetnik approved these changes
@s3ths1 s3ths1 Awaiting requested review from s3ths1
@borisalekseev borisalekseev Awaiting requested review from borisalekseev
@draincoder draincoder Awaiting requested review from draincoder

Assignees

@powersemmi powersemmi

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation github_actions Pull requests that update GitHub Actions code Redis Issues related to `faststream.redis` module and Redis features

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature: Adds support for the Redis cluster

3 participants

@powersemmi @fbartonoctopus @Lancetnik