[CELEBORN-2468] Support pluggable transport authentication and authorization by Kalvin2077 · Pull Request #3857 · apache/celeborn

Kalvin2077 · 2026-09-23T07:59:48Z

A large portion of the new lines of code are test code, but the feature involves quite a few files.

What changes were proposed in this pull request?

Add configurable transport authentication and connection-bound authorization across Celeborn's built-in RPC, shuffle data and replication paths:

Load ordered TransportClientBootstrap and TransportServerBootstrap implementations from exact-module configuration, and manage configured instances through TransportContext startup and shutdown.
Add a ConnectionSecurityContext policy for operation-specific application/service access and effective-user resolution. Keep plugin identity separate from native clientId; make native-policy composition explicit.
Preserve the receiving connection and local/remote origin through request/reply and one-way RPC dispatch, and apply checks before protected Master, Worker and LifecycleManager operations.
Authorize push, replication and stream operations before their side effects, using server-held resource state for subsequent stream requests.
Document deployment, lifecycle and composition contracts, and add external-JAR, native-composition, handler and replicated-shuffle tests.

Why are the changes needed?

Deployments use different identity providers, credentials and authorization policies. Bootstrap interfaces alone do not provide configurable loading, lifecycle ownership and consistent request authorization across Celeborn's transports. A connection-bound policy lets an external mechanism retain its verified identity and participate in resource and accounting decisions without spreading provider-specific code through the OSS handlers.

Reply semantics do not determine whether an operation needs authorization: one-way RPCs and stream controls can also change protected state. Both paths need the receiving connection and the relevant resource facts before business handling.

Does this PR resolve a correctness bug?

Yes

A chunk fetch for a stream that is no longer registered previously dereferenced streamState.buffers before checking whether streamState was null. It now returns ChunkFetchFailure. The focused regression test fails with the old implementation and passes with this change.

Does this PR introduce any user-facing change?

Yes

Adds celeborn.<module>.client.bootstrap.classes and celeborn.<module>.server.bootstrap.classes, both empty by default, plus plugin authorization APIs and deployment guidance. Configured bootstraps also run when native authentication is disabled. Configured handshakes precede native registration/SASL where those are enabled. Installing a plugin policy replaces native authorization; a policy that also requires native restrictions calls them explicitly.

Built-in request paths now apply operation-specific checks before processing protected requests. Public and internal RPC listeners still share rpc_service configuration. The policy request does not expose stream ID or creator channel. Native mode does not add SASL to internal RPC or replication connections; configured bootstraps can authenticate those peers. Custom RPC endpoints must implement their own authorization hook.

How was this patch tested?

Unit and integration tests passed for bootstrap lifecycle, native-policy composition, RPC authorization (including one-way messages), and push/replication/stream access checks.
External-JAR loading, shaded-client integration and replicated-shuffle tests passed. The missing-stream regression test fails on the old implementation and passes with this patch.
The OSS/native benchmark completed 16 Spark applications and 80 integrity-checked rounds on one Master and two Workers (4 vCPUs each), using Spark 3.5.3 and Java 8. Replication was enabled and compression disabled.
The table compares candidate versus baseline under the same authentication mode. Values are mean paired changes from two applications per mode/workload, each with two warmup and three measured rounds. These descriptive results do not establish statistical significance; CPU sampling uncertainty remains.

gauravkm · 2026-09-23T16:58:16Z

CI has six failing service checks that need investigation: Java 8 / Scala 2.12.18 and 2.13.16, Java 11 / Scala 2.12.18 and 2.13.16, and Java 17 / Scala 2.12.18 and 2.13.16. The Java 11 / Scala 2.13 log reports a StoragePolicyCase1 failure.

The code review looks good; no introduced P0/P1 issues found. I checked bootstrap ordering and failure handling, native SASL compatibility, request/reply authorization, and verified quota/resource ownership. Added tests cover authentication rejection, cross-application access, reconnects, and Flink context cleanup.

Copilot

Copilot review overview

🟡 Changes recommended

Critical authorization gaps remain for PbWorkerLost and PbCheckWorkersAvailable paths.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

Worker availability RPC bypasses service authorization · New
Unauthenticated one-way worker-loss messages bypass authorization · New
Remote one-way worker-loss requests bypass service-role checks · New

What changed in this PR

Adds configurable transport authentication/authorization hooks, verified identity propagation, and client lifecycle updates.

Changes:

Adds per-module transport bootstrap loading and cleanup.
Introduces application and service authorization policies.
Adds security coverage across RPC, Master, Worker, and Flink integrations.

File	Description
`worker/src/test/scala/org/apache/celeborn/service/deploy/worker/ControllerSecuritySuite.scala`	Tests verified worker ownership.
`worker/src/test/scala/org/apache/celeborn/service/deploy/cluster/AuthenticatedShuffleSuite.scala`	Tests authenticated shuffle operations.
`worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Worker.scala`	Closes replication resources.
`worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala`	Uses verified identities.
`master/src/test/scala/org/apache/celeborn/service/deploy/master/MasterSuite.scala`	Updates transport test fixtures.
`master/src/test/scala/org/apache/celeborn/service/deploy/master/MasterSecuritySuite.scala`	Tests service policies and quota identity.
`master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala`	Enforces service policies and verified identities.
`common/src/test/scala/org/apache/celeborn/common/rpc/RpcEndpointSecuritySuite.scala`	Tests RPC security policies.
`common/src/test/scala/org/apache/celeborn/common/rpc/RpcApplicationAuthorizationSuite.scala`	Tests application authorization.
`common/src/test/resources/auth-plugin/TokenBootstrap.java`	Provides an external authentication fixture.
`common/src/test/java/org/apache/celeborn/common/network/security/TransportBootstrapSuiteJ.java`	Tests bootstrap loading and cleanup.
`common/src/test/java/org/apache/celeborn/common/network/security/ExternalAuthenticationSuiteJ.java`	Tests external authentication flows.
`common/src/main/scala/org/apache/celeborn/common/rpc/RpcEndpoint.scala`	Adds authorization and identity helpers.
`common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala`	Defines bootstrap configuration.
`common/src/main/java/org/apache/celeborn/common/network/util/TransportConf.java`	Exposes bootstrap configuration.
`common/src/main/java/org/apache/celeborn/common/network/TransportContext.java`	Loads and manages bootstraps.
`common/src/main/java/org/apache/celeborn/common/network/server/TransportServerBootstrap.java`	Adds server bootstrap lifecycle support.
`common/src/main/java/org/apache/celeborn/common/network/server/TransportServer.java`	Applies configured server bootstraps.
`common/src/main/java/org/apache/celeborn/common/network/server/BaseMessageHandler.java`	Delegates application authorization.
`common/src/main/java/org/apache/celeborn/common/network/security/ServiceAuthorizer.java`	Defines service authorization operations.
`common/src/main/java/org/apache/celeborn/common/network/security/ApplicationAuthorizer.java`	Defines application authorization.
`common/src/main/java/org/apache/celeborn/common/network/client/TransportClientFactory.java`	Applies configured client bootstraps.
`common/src/main/java/org/apache/celeborn/common/network/client/TransportClientBootstrap.java`	Adds client bootstrap lifecycle support.
`common/src/main/java/org/apache/celeborn/common/network/client/TransportClient.java`	Stores connection policies and identity.
`client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java`	Delays secured transport initialization.
`client-flink/common/src/test/java/org/apache/celeborn/plugin/flink/FlinkAuthenticationSuiteJ.java`	Tests Flink bootstrap cleanup.
`client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/client/FlinkShuffleClientImpl.java`	Defers transport initialization and manages cleanup.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

gauravkm · 2026-09-29T03:25:54Z

CI currently has failing checks: flink2 (11, 2.2); service (11); service (8); service (17). Other checks are still running.

I revisited the current revision and am withholding a positive code-review conclusion. The earlier “Looks good” comment should not be treated as clearance for this revision.

gauravkm · 2026-09-30T05:54:58Z

CI is still failing: Maven service (8), service (11), and service (17). The Java 11 log shows a ConfigurationSuite golden-file mismatch for docs/configuration/network.md (trailing whitespace); I have not established the causes of the other two failures. Other checks are still running.

Looks good from my code review of ef31bbb; no P0/P1 issues found. I checked remote versus local dispatch, internal forwarding, native/custom authentication composition, effective quota/ownership identity, replica permissions, stream controls and bootstrap cleanup. The shared dispatch hook applies policy before both request/reply and one-way processing. Added test source covers real RPC denial, worker-loss state preservation, reconnects and replicated shuffle data/ownership. This code-review conclusion does not clear the outstanding CI failures.

gauravkm · 2026-09-30T06:47:57Z

CI update for ef31bbb: six additional SBT service checks have now failed: Java 8 / Scala 2.12.18 and 2.13.16, Java 11 / Scala 2.12.18 and 2.13.16, and Java 17 / Scala 2.12.18 and 2.13.16. The three Maven service failures in my previous summary remain. I have not established the causes of these additional failures; the earlier Java 11 Maven log diagnosis does not establish their cause.

gauravkm · 2026-09-30T08:52:30Z

CI on 28d44d5 has eight failing checks: Maven service Java 8, 11, 17; SBT service Java 11 / Scala 2.12.18, 2.13.16, Java 17 / Scala 2.12.18, 2.13.16; and Spark 3.4 / Java 8 / LocalDiskShuffleDataIO. Their causes remain unverified; the previous revision’s whitespace diagnosis does not explain these failures.

Looks good from my code review; no P0/P1 issues found. I verified that this revision adds only two trailing spaces to the generated configuration table, with all production and test files unchanged. The previously checked RPC authorization, identity propagation, replication and stream controls—and their test coverage—remain intact. CI investigation is still outstanding.

gauravkm · 2026-09-30T10:51:18Z

CI update for 28d44d5: Spark 3.5 / Java 8 / LocalDiskShuffleDataIO has also failed. The run now has nine failing checks, including the eight listed above, and no checks still running. I have not established the cause of this additional failure.

Initialize and clean up MemoryManager per test to avoid failures caused by suite discovery and execution order. Forward parent JVM module-access options to worker startup probes. Use a bounded wait for asynchronous NIO socket release while retaining the port reuse and resource cleanup assertions.

gauravkm · 2026-10-02T02:28:20Z

CI on a3e730a has two failing checks: flink2 (17, 2.2) and flink2 (11, 2.0). The Java 17/Flink 2.2 log reports RemoteShuffleMasterSuiteJV22.testRegisterPartitionWithProducer failing its counter assertion at line 145: expected 2, observed 1. I have not established the root causes of these failures. Other checks are still running.

Looks good from my code review; no P0/P1 issues found. Production authorization code is unchanged from 28d44d5, and the prior dispatch fixes remain intact. I checked the child-JVM module flags, bounded listener-rebind assertion and per-test MemoryManager cleanup; the startup-failure, close-count and storage-tier assertions remain. This code-review conclusion does not clear the outstanding CI failures.

gauravkm · 2026-10-02T03:25:01Z

CI update for a3e730a: SBT service (Java 8 / Scala 2.12.18) has also failed. Its log lists GrpcRatisMasterStatusSystemSuiteJ under failed tests. I have not established the cause. The two Flink failures in my previous summary remain, bringing the current total to three failing checks; one Spark check is still running.

Kalvin2077 · 2026-10-08T10:48:24Z

PTAL. @SteNicholas @zaynt4606 @RexXiong

github-actions Bot added module:client module:common module:master module:worker module:flink correctness Correctness bugfix labels Sep 23, 2026

SteNicholas requested a lite review from Copilot September 24, 2026 08:42

Copilot started reviewing on behalf of SteNicholas September 24, 2026 08:43 View session

Copilot AI reviewed Sep 24, 2026

View reviewed changes

feat(security): bind authorization policy to transport connections

a203e6b

Kalvin2077 force-pushed the refactor/security-framework branch from 9f88255 to a203e6b Compare September 29, 2026 02:24

github-actions Bot removed module:client module:master module:worker module:flink labels Sep 29, 2026

feat(network): load and own configured transport bootstraps

8ac7ddf

github-actions Bot added module:client kind:documentation module:worker module:flink labels Sep 29, 2026

gauravkm reviewed Sep 29, 2026

View reviewed changes

Comment thread common/src/main/java/org/apache/celeborn/common/network/client/TransportClient.java

Kalvin2077 added 7 commits September 29, 2026 15:23

feat(security): compose external authentication with native SASL

bad4e17

feat(rpc): preserve authorization context for ask and send

8448b93

feat(master): authorize control operations and resolve users

b72edce

feat(security): authorize worker and lifecycle control requests

9db7b11

feat(worker): authorize shuffle data and stream operations

031c91d

refactor(security): remove redundant context and stream restrictions

5aba5a9

test(security): verify replicated shuffle and document plugin use

ef31bbb

github-actions Bot added the module:master label Sep 30, 2026

Kalvin2077 changed the title ~~[CELEBORN-2468] Add transport security extension points for custom authn and authz~~ [CELEBORN-2468] Support pluggable transport authentication and authorization Sep 30, 2026

fix: doc format

28d44d5

test(flink): fix heartbeat races in shuffle counter assertions

cb28137

Navigation Menu

Repository navigation

[CELEBORN-2468] Support pluggable transport authentication and authorization - #3857

[CELEBORN-2468] Support pluggable transport authentication and authorization#3857
Kalvin2077 wants to merge 12 commits into
apache:mainfrom
Kalvin2077:refactor/security-framework

Kalvin2077 commented Sep 23, 2026 •

edited

Loading

Uh oh!

gauravkm commented Sep 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gauravkm commented Sep 29, 2026

Uh oh!

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Oct 2, 2026

Uh oh!

gauravkm commented Oct 2, 2026

Uh oh!

Kalvin2077 commented Oct 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Kalvin2077 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR resolve a correctness bug?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gauravkm commented Sep 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gauravkm commented Sep 29, 2026

Uh oh!

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Sep 30, 2026

Uh oh!

gauravkm commented Oct 2, 2026

Uh oh!

gauravkm commented Oct 2, 2026

Uh oh!

Kalvin2077 commented Oct 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kalvin2077 commented Sep 23, 2026 •

edited

Loading