feat(server)!: auth-gate cluster metadata, forward register to primary by hubcio · Pull Request #3872 · apache/iggy

hubcio · 2026-08-12T12:29:40Z

Unauthenticated clients could read the full cluster roster through
the binary GetClusterMetadata pre-auth carve-out, leaking private
network topology. Plain gating would strand any client that dialed
a backup: Register commits only on the primary, and without the
pre-auth read there is no leader discovery left, so login replays
transient denies until the read budget dies.

Gate the read on every roster shape (PING stays the only pre-auth
code) and let the dialed backup complete the login instead:
credentials are verified locally against replicated user state,
only the verified identity (client id, user id) crosses the
replica interconnect as a sealed ForwardRegister frame, the
primary commits the Register, and the backup binds the session
and replies on its own connection. No credentials travel on the
wire, no reply routing lands on the primary, and the HTTP
forward's fail-closed semantics (an unreplicated PAT is denied at
the backup) carry over; a forward that never left the backup maps
to the retryable class, not a client-side 503. An unbound reader
is denied with a plain Unauthenticated Reply on the status
channel, never a session-terminal Eviction, so a foreign SDK's
stray probe cannot tear down the connection its login is about to
use. The Rust and Node SDKs drop their pre-login leader probe;
the authenticated post-login redirect is the sole leader
settlement, which Node gains here with a short poll through an
election. The C++ e2e expectation of the pre-auth read inverts to
match.

Kill-primary testing of the forward exposed a pre-existing
view-start bug: the metadata parked-view arm dispatched
RebuildPipeline behind the superblock persist, so a login admitted
during that fsync minted an op into the still-empty pipeline and
the deferred rebuild panicked the pump ("sequence must be
sequential"), leaving a deaf primary holding its quorum slot while
the survivor spun elections forever. Dispatch the local actions
before the persist, as every sibling site already does. The drain
path now surfaces the pump's panic verdict instead of claiming a
clean exit, the harness fails any test whose server wrote a panic
to stderr, a simulator test pins the interleaving
deterministically, and forward failures no longer masquerade as a
view-change cancel.

codecov · 2026-08-12T12:32:21Z

Codecov Report

❌ Patch coverage is 91.02857% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.05%. Comparing base (7dee4d8) to head (da26b83).

Files with missing lines	Patch %	Lines
core/binary_protocol/src/consensus/header.rs	90.15%	25 Missing and 14 partials ⚠️
core/server/src/dispatch.rs	94.67%	36 Missing and 3 partials ⚠️
core/server/src/bootstrap.rs	62.16%	12 Missing and 2 partials ⚠️
core/server_common/src/consensus_message.rs	70.21%	8 Missing and 6 partials ⚠️
...ache/iggy/client/async/tcp/AsyncIggyTcpClient.java	52.17%	9 Missing and 2 partials ⚠️
foreign/go/client/tcp/tcp_session_management.go	67.85%	6 Missing and 3 partials ⚠️
foreign/node/src/client/client.socket.ts	92.74%	9 Missing ⚠️
foreign/go/internal/util/leader_aware.go	84.78%	5 Missing and 2 partials ⚠️
core/server/src/http/state.rs	84.61%	1 Missing and 3 partials ⚠️
core/sdk/src/leader_aware.rs	88.23%	2 Missing ⚠️
... and 6 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3872      +/-   ##
============================================
- Coverage     82.69%   82.05%   -0.64%     
- Complexity     1296     1299       +3     
============================================
  Files          1199     1199              
  Lines        160319   161885    +1566     
  Branches     129924   131465    +1541     
============================================
+ Hits         132570   132836     +266     
- Misses        24235    25428    +1193     
- Partials       3514     3621     +107

Components	Coverage Δ
Rust Core	`83.30% <92.23%> (+0.12%)`	⬆️
Java SDK	`66.06% <54.16%> (-0.09%)`	⬇️
C# SDK	`60.58% <50.00%> (-15.09%)`	⬇️
Python SDK	`89.98% <ø> (ø)`
PHP SDK	`84.26% <ø> (ø)`
Node SDK	`96.21% <92.74%> (-0.04%)`	⬇️
Go SDK	`68.53% <78.94%> (-0.08%)`	⬇️

Files with missing lines	Coverage Δ
core/binary_protocol/src/consensus/command.rs	`100.00% <100.00%> (ø)`
core/sdk/src/quic/quic_client.rs	`75.07% <100.00%> (+0.22%)`	⬆️
core/sdk/src/tcp/tcp_client.rs	`76.17% <100.00%> (-0.08%)`	⬇️
core/sdk/src/websocket/websocket_client.rs	`73.34% <100.00%> (+0.04%)`	⬆️
core/server/src/auth.rs	`95.06% <100.00%> (+0.59%)`	⬆️
core/server/src/server_error.rs	`61.40% <ø> (ø)`
core/server/src/session_manager.rs	`90.37% <ø> (+0.34%)`	⬆️
core/shard/src/router.rs	`85.13% <100.00%> (-0.62%)`	⬇️
core/simulator/src/deps.rs	`73.71% <100.00%> (+0.83%)`	⬆️
core/simulator/src/executor/mod.rs	`90.45% <100.00%> (ø)`
... and 18 more

... and 82 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Unauthenticated clients could read the full cluster roster through the binary GetClusterMetadata pre-auth carve-out, leaking private network topology. Plain gating would strand any client that dialed a backup: Register commits only on the primary, and without the pre-auth read there is no leader discovery left, so login replays transient denies until the read budget dies. Gate the read on every roster shape (PING stays the only pre-auth code) and let the dialed backup complete the login instead: credentials are verified locally against replicated user state, only the verified identity (client id, user id) crosses the replica interconnect as a sealed ForwardRegister frame, the primary commits the Register, and the backup binds the session and replies on its own connection. No credentials travel on the wire, no reply routing lands on the primary, and the HTTP forward's fail-closed semantics (an unreplicated PAT is denied at the backup) carry over; a forward that never left the backup maps to the retryable class, not a client-side 503. An unbound reader is denied with a plain Unauthenticated Reply on the status channel, never a session-terminal Eviction, so a foreign SDK's stray probe cannot tear down the connection its login is about to use. The Rust, Node, Go, and C# SDKs drop their pre-login leader probe; the authenticated post-login redirect is the sole leader settlement. Node and Go gain it here with a short poll through an election (Go replays its sign-in before settling so the leader connection is authenticated), C# already settled post-login and only sheds the dead probe. The C++ e2e expectation of the pre-auth read inverts to match, and the C# PAT redirection test polls through the replication window that a backup's fail-closed PAT verify leaves open instead of asserting on its width. Kill-primary testing of the forward exposed a pre-existing view-start bug: the metadata parked-view arm dispatched RebuildPipeline behind the superblock persist, so a login admitted during that fsync minted an op into the still-empty pipeline and the deferred rebuild panicked the pump ("sequence must be sequential"), leaving a deaf primary holding its quorum slot while the survivor spun elections forever. Dispatch the local actions before the persist, as every sibling site already does. The drain path now surfaces the pump's panic verdict instead of claiming a clean exit, the harness fails any test whose server wrote a panic to stderr, a simulator test pins the interleaving deterministically, and forward failures no longer masquerade as a view-change cancel.

Unauthenticated clients could read the full cluster roster through the binary GetClusterMetadata pre-auth carve-out, leaking private network topology. Plain gating would strand any client that dialed a backup: Register commits only on the primary, and without the pre-auth read there is no leader discovery left, so login replays transient denies until the read budget dies. Gate the read on every roster shape (PING stays the only pre-auth code) and let the dialed backup complete the login instead: credentials are verified locally against replicated user state, only the verified identity (client id, user id) crosses the replica interconnect as a sealed ForwardRegister frame, the primary commits the Register, and the backup binds the session and replies on its own connection. No credentials travel on the wire, no reply routing lands on the primary, and the HTTP forward's fail-closed semantics (an unreplicated PAT is denied at the backup) carry over; a forward that never left the backup maps to the retryable class, not a client-side 503. An unbound reader is denied with a plain Unauthenticated Reply on the status channel, never a session-terminal Eviction, so a foreign SDK's stray probe cannot tear down the connection its login is about to use. The Rust, Node, Go, and C# SDKs drop their pre-login leader probe; the authenticated post-login redirect is the sole leader settlement. Node and Go gain it here with a short poll through an election (Go replays its sign-in before settling so the leader connection is authenticated), C# already settled post-login and sheds the dead probe. The C# settle also compares the leader against the endpoint the socket resolved rather than the configured string: a client configured with a hostname was never "on" the leader the roster names by IP, so every login reconnected it to the node it was already talking to - masked until now by the pre-login probe rewriting the address first. The C++ e2e expectation of the pre-auth read inverts to match, the C# PAT redirection test polls through the replication window that a backup's fail-closed PAT verify leaves open instead of asserting on its width, and the C# BDD same-server step compares resolved endpoints, since a never-redirected client keeps the hostname it was given while a redirected one holds the roster address. Kill-primary testing of the forward exposed a pre-existing view-start bug: the metadata parked-view arm dispatched RebuildPipeline behind the superblock persist, so a login admitted during that fsync minted an op into the still-empty pipeline and the deferred rebuild panicked the pump ("sequence must be sequential"), leaving a deaf primary holding its quorum slot while the survivor spun elections forever. Dispatch the local actions before the persist, as every sibling site already does. The drain path now surfaces the pump's panic verdict instead of claiming a clean exit, the harness fails any test whose server wrote a panic to stderr, a simulator test pins the interleaving deterministically, and forward failures no longer masquerade as a view-change cancel.

…/login-forward

Copilot

Pull request overview

This PR hardens cluster topology privacy by auth-gating GetClusterMetadata across transports, while preserving usability by forwarding Register/Logout from backups to the metadata primary and moving leader settlement to post-login across SDKs. It also fixes a view-start ordering bug that could corrupt the metadata pipeline during superblock persist, and improves test harnesses to fail on shard pump panics that previously went unnoticed.

Changes:

Auth-gate GetClusterMetadata, add replica-interconnect frames for forwarded Register/Logout, and update Rust/Node/Go/Java/C#/C++ SDK behavior to settle leadership after a successful login.
Fix metadata view-start action ordering (dispatch local actions before persist) and strengthen shutdown drain handling to surface shard pump failure/timeout as errors.
Add deterministic simulator + cluster integration coverage for the failover/login-forwarding interleavings; bump versions across crates/SDKs/UI/connectors.

Reviewed changes

Copilot reviewed 87 out of 93 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
web/package.json	Web UI version bump.
web/package-lock.json	Web UI lockfile version bump.
foreign/python/uv.lock	Python SDK lockfile version bump.
foreign/python/pyproject.toml	Python SDK version bump.
foreign/python/Cargo.toml	Python Rust crate version bump + iggy SDK dep bump.
foreign/node/src/client/client.socket.ts	Node TCP client: remove pre-auth roster probe; post-login leader settlement with bounded polling/redirects.
foreign/node/src/client/client.socket.test.ts	Node tests updated/extended for post-login leader settlement + leaderless polling behavior.
foreign/node/package.json	Node SDK version bump.
foreign/node/package-lock.json	Node SDK lockfile version bump.
foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/tcp/AsyncTcpConnectionConcurrencyTest.java	Java async TCP concurrency test updated to use ping as only pre-auth probe.
foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/tcp/AsyncIggyTcpClientLoginRoutingTest.java	Java login routing semantics updated (login replay per redirect).
foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/AsyncConnectionPoolAuthTest.java	Java: assert only ping allowed pre-login; cluster metadata now rejected before connect/login.
foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/AsyncTcpConnection.java	Java TCP: only ping allowed pre-auth; roster auth-gated.
foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/AsyncIggyTcpClient.java	Java: login + authenticated leader settlement serialized across concurrent logins.
foreign/go/tests/e2e_test.go	Go e2e: expect unauthenticated `GetClusterMetadata` to fail; clarify pre-auth policy.
foreign/go/internal/util/leader_aware.go	Go: poll leaderless rosters for a bounded budget; handle unauthenticated roster read as “stay put”.
foreign/go/internal/util/leader_aware_test.go	Go: tests for leaderless polling budget behavior.
foreign/go/contracts/version.go	Go SDK version bump constant.
foreign/go/client/tcp/tcp_session_management.go	Go TCP: sign-in then authenticated leader settlement; replay sign-in on redirects.
foreign/go/client/tcp/tcp_core.go	Go TCP: remove pre-login leadership settlement; adjust connect-scoped semantics and comments accordingly.
foreign/go/client/tcp/tcp_core_review_test.go	Go TCP: update connect/sign-in suppression expectations.
foreign/go/client/tcp/tcp_connect_test.go	Go TCP: update connect tests to sign-in first, then read roster (authenticated) for settlement.
foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.Vsr.cs	C# VSR: compare leader endpoint against resolved remote endpoint to avoid hostname/IP mismatch redirects.
foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.cs	C# TCP: track remote endpoint; remove pre-login roster redirect; settle after login.
foreign/csharp/Iggy_SDK/Iggy_SDK.csproj	C# SDK version bump.
foreign/csharp/Iggy_SDK.Tests.Integration/IggyTlsConnectionTests.cs	C# integration: adjust expected failure mode now that connect sends nothing pre-login.
foreign/csharp/Iggy_SDK.Tests.Integration/ClusterRedirectionTests.cs	C# integration: PAT login on follower may fail until replication; poll through window.
foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/LeaderRedirectionSteps.cs	C# BDD: compare resolved endpoints (hostname vs IP) for “same server” assertions.
foreign/cpp/tests/e2e/client.cpp	C++ e2e: `GetClusterMetadata` now rejected pre-login; allowed post-login.
examples/python/uv.lock	Python examples lockfile bump.
bdd/python/uv.lock	Python BDD lockfile bump.
core/simulator/src/lib.rs	Add deterministic simulator regression test for view-start persist interleaving + register submission.
core/simulator/src/executor/mod.rs	Expose `yield_once` helper for deterministic interleavings; remove duplicate test-local version.
core/simulator/src/deps.rs	Add superblock write yield fault injection (`set_yield_writes`) using `yield_once`.
core/shard/src/router.rs	Route new forward frame variants to metadata group; document operational impact of forward drops.
core/shard/src/lib.rs	Add forward nonce + parking maps; handle forward frames; dispatch local VSR actions before persist to avoid pipeline corruption.
core/server/src/session_manager.rs	Update roster comment (no longer “pre-auth carve-out”).
core/server/src/server_error.rs	Add shard pump drain timeout/death error variants.
core/server/src/http/state.rs	Factor register-submit error mapping; add coverage for preserving known/unknown outcomes.
core/server/src/dispatch/authz.rs	Add unbound deny reply path (no commit frontier leakage) and gate `GET_CLUSTER_METADATA`.
core/server/src/bootstrap.rs	Make pump drain failures/timeout error out; plumb panic reasons into structured errors; add unit test for drain-timeout reporting.
core/server/src/auth.rs	Ensure login replies don’t report commit behind newly bound session; refine transient replay codes for unknown outcomes.
core/server/Cargo.toml	Server crate version bump.
core/server_common/src/consensus_message.rs	Add forward frame variants to `MessageBag` and parsing/round-trip tests.
core/sdk/src/websocket/websocket_client.rs	Rust WS transport: prevent reconnect loop on unauthenticated roster probe; settle leadership post-login.
core/sdk/src/tcp/tcp_client.rs	Rust TCP transport: same unauthenticated probe rule + post-login leader settlement behavior.
core/sdk/src/quic/quic_client.rs	Rust QUIC transport: same unauthenticated probe rule + post-login leader settlement behavior.
core/sdk/src/leader_aware.rs	Add shared detection of unauthenticated `GetClusterMetadata` probes; tolerate post-login Unauthenticated as “session died”.
core/sdk/Cargo.toml	Rust SDK crate version bump.
core/metadata/src/impls/metadata.rs	Add forward-specific submit errors (primary unreachable / forward timeout) + document transient policy.
core/integration/tests/server/scenarios/authentication_scenario.rs	Integration: stop skipping `GET_CLUSTER_METADATA` in auth-required checks.
core/integration/tests/server/legacy_login_vsr.rs	Update legacy VSR login expectations now that pre-auth guard denies with Unauthenticated reply.
core/integration/tests/cluster/register_forwarding.rs	New integration tests: login/logout forwarding via backup, PAT replication behavior, and primary-kill scenarios.
core/integration/tests/cluster/mod.rs	Register new forwarding test module.
core/integration/tests/cluster/client_table_restart.rs	Keep authenticated attempts alive across retries to avoid cleanup races post-forwarding.
core/integration/src/harness/handle/server.rs	Fail tests on server panics detected in stderr; print logs for diagnosis.
core/integration/src/harness/handle/common.rs	Add stderr panic report extraction (with blocking-pool panic filtering).
core/connectors/sources/random_source/Cargo.toml	Connector version bump.
core/connectors/sources/postgres_source/Cargo.toml	Connector version bump.
core/connectors/sources/influxdb_source/Cargo.toml	Connector version bump.
core/connectors/sources/elasticsearch_source/Cargo.toml	Connector version bump.
core/connectors/sinks/surrealdb_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/stdout_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/s3_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/quickwit_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/postgres_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/mongodb_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/meilisearch_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/influxdb_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/iceberg_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/http_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/elasticsearch_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/doris_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/delta_sink/Cargo.toml	Connector version bump.
core/connectors/sinks/clickhouse_sink/Cargo.toml	Connector version bump.
core/connectors/sdk/Cargo.toml	Connector SDK version bump (breaking bump aligns with auth/forwarding changes).
core/connectors/runtime/Cargo.toml	Connectors runtime version bump.
core/common/src/traits/cluster_client.rs	Document `get_cluster_metadata` as authenticated-only.
core/common/Cargo.toml	Core common crate version bump.
core/cli/Cargo.toml	CLI crate version bump.
core/binary_protocol/src/lib.rs	Re-export new forward frame header types.
core/binary_protocol/src/consensus/mod.rs	Re-export new consensus header types for forwarding frames.
core/binary_protocol/src/consensus/header.rs	Define and validate forward register/logout headers + results; document rolling-upgrade incompatibility.
core/binary_protocol/src/consensus/command.rs	Add command discriminants 26–29; update bit-pattern validity tests.
core/binary_protocol/Cargo.toml	Binary protocol crate version bump.
core/bench/report/Cargo.toml	Bench report crate version bump.
core/bench/dashboard/server/Cargo.toml	Bench dashboard server version bump.
core/bench/dashboard/frontend/Cargo.toml	Bench dashboard frontend version bump.
core/bench/Cargo.toml	Bench crate version bump.
core/ai/mcp/Cargo.toml	MCP crate version bump.
Cargo.toml	Workspace dependency version bumps (iggy, cli, protocol, common, connector sdk).
Cargo.lock	Lockfile updated for version bumps.

Files not reviewed (2)

foreign/node/package-lock.json: Generated file
web/package-lock.json: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 12, 2026

hubcio force-pushed the feat/login-forward branch 3 times, most recently from 79d021d to c563145 Compare August 12, 2026 14:00