Repository navigation
Conversation
Lay the foundation for sharing a `signing` secret across a horizontally
scaled enclave fleet: the master holds the full secret and acts as a
threshold-sdk leader, and same-image joiner enclaves attest in to receive
their shares. Untagged/static secrets are unchanged.
Config:
- Add `kind` to secrets (static|signing) with validation, thread it through
the CLI build config and enclave.yaml scaffolding.
Runtime:
- ScalingEntity (scaling.go): role-gated leader/follower, attested admission
handshake (/enclave/handshake/{nonce,admit}) with mutual PCR verification,
one-shot nonces, and a NaCl-sealed reply; loads/generates the threshold
host key via KMS. Relay config is port-only (ENCLAVE_SCALING_LISTEN_PORT /
ENCLAVE_SCALING_LEADER_ADDR), bound on the TAP address.
- DkgStore (chilldkg.go): thresholdsdk.Store over encrypted Storage; persists
only public recovery blobs and returns the master's KMS secret as a0, so the
reconstructed group secret equals the normally-created secret.
- nitriding.ArePCRsIdenticalForKeys: compare only code-identity PCRs (0,1,2)
so same-image enclaves on different EC2 instances admit each other.
- KMS.decryptToBytes: attestation-gated decrypt to raw plaintext.
- StaticSecrets: wire the scaling subsystem, add the follower key-derivation
path, and skip signing secrets in ExtendPCRs.
- Build ScalingEntity in Init when ENCLAVE_THRESHOLD_ROLE is set; mount the
leader admission routes on the public listener.
Wire the scaling subsystem end to end so leader and follower enclaves hold and reload their threshold signing material across reboots. - Rename ENCLAVE_THRESHOLD_ROLE -> ENCLAVE_SCALING_ROLE (code + design doc). - Persist derived shares: both roles drain the SDK Keys() channel into StoreDerivedSecret, which bundles the master (a0) and this node's share into one KMS-encrypted envelope per secret. A later boot loads both from the single SSM slot — EnvVar=master + ShareEnvVar=share on a leader, EnvVar=share on a follower — instead of re-running a ceremony. - Pin the group key: DkgStore.GetLeaderSecret resolves a0 from the master's env var, so the group forms around the KMS-managed secret rather than a freshly generated one. Drop the no-op PutLeaderSecret (the SDK now owns a0's lifecycle through GetLeaderSecret alone). - Wire ScalingEntity.kms and move scaling Init after EnsureKeyID so the KMS-backed host key can be generated. - Build the scaling entity in New so a leader's admission handshake routes actually mount (gated on IsLeader); AWS-backed bring-up stays in Init.
…e scaling ceremony Builds on the persisted-share scaffolding to run a 1-leader + N-follower threshold ceremony across single-host enclaves, with re-randomized shares taking effect in the already-running app. Runtime: - Re-fork the child app when a reshare re-materializes its share. A share reaches the app only through the env var it inherits at fork, so a post-init reshare isn't visible live; the share handler now signals main to restart the app once it has started. Collapse the app lifecycle into one upstreamState (NotStarted -> Running -> Exited), replacing the started/exited booleans. - Initialize Storage before the scaling entity, so the leader can persist DKG recovery blobs the instant a follower joins (was initialized last, racing the handshake coming up). - Construct the scaling entity in Init from the SSM-overlaid role and mount the leader handshake routes unconditionally, gated per-request by readyLeader. - Reshare only "signing" secrets on a follower; keep the leader master (a0) in an in-memory cache so it survives once the env var holds the node's share. Drop ShareEnvVar -- one env var per secret. - Skip TLS PKI on the follower->leader handshake (trust is mutual NSM attestation) and read local PCRs without strict verification in dev. - Add a per-instance kernel-cmdline override channel (devcmdline) so same-image nodes get their deployment / role / vsock ports without changing PCRs. Test: - run-scaling.sh boots the ceremony on one host (per-node vsock CID, gvproxy port, SSM namespace); scaling-test.sh asserts three distinct valid shares and that a new join re-randomizes an incumbent. seed-scaling.sh provisions the per-deployment KMS placeholder + storage bucket (mirroring tofu). Adds `make test-scaling` + a docker-compose scaling-runner; /test/scaling exposes each node's share pubkey.
…aling CI
Replace the machine-local 'replace => /home/...' on the threshold SDK/core with versioned requires on the now-public modules github.com/ArkLabsHQ/threshold-magic/threshold_{sdk,core}@v0.1.0, so the runtime module resolves anywhere (CI, nix, fresh checkouts) instead of only on one dev's machine.
Add .github/workflows/scaling-test.yml: boots the 1-leader + 2-follower ceremony on ubuntu-latest (KVM + vhost-device-vsock), mirroring 'make test-scaling'. Validation-phase triggers (workflow_dispatch + push:[dkg]) until proven green on GitHub's kernel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…improve related comments - enhance metrics struct for clarity and consistency - adjust scaling logic to reference leader secret instead of master test(runtime): modify tests to reflect changes in leader secret handling docs: add Threshold Scaling Architecture documentation
…/follower system details
Add autonomous membership revocation to the threshold-scaling cluster. The leader continuously monitors follower liveness and automatically evicts failed members, triggering a resharing of all secrets among the remaining quorum. This closes the "removal requires a leave-triggered reshare" gap documented in SCALING-ARCHITECTURE.md. Mechanism: - Followers periodically sign a fresh leader nonce with their threshold host key (BIP-340 Schnorr) and POST it to /enclave/heartbeat. - The leader verifies the signature and nonce, recording each follower's last-seen timestamp. - A background monitor evicts followers that exceed ENCLAVE_SCALING_LIVENESS_TIMEOUT. - Eviction invokes Leader.Ban, removing the member, rekeying all secrets, and invalidating the follower's share while preserving the threshold public key. - Bans are persisted in encrypted storage and enforced during both authorization and attested admission, preventing re-admission across leader restarts. Heartbeat intervals and liveness timeouts are configurable via the enclavecfg.* command-line whitelist. Also simplifies the follower join path by inlining followerJoinHandshake into startFollower, and adds comprehensive unit and end-to-end tests covering heartbeat authentication, automatic eviction, durable ban enforcement, and resharing after follower failure.
|
This PR has been open for 34 days without a review decision. @aruokhai is anyone looking at this? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Review — Leader/Follower Static Secret Sharing (DKG)
Summary
Adds threshold cryptography for enclave signing: a signing secret is shared amongst N enclaves, with a quorum required to sign, while the group public key stays pinned to the original secret across resharing.
Areas requiring human expert review
-
DKG correctness — Uses
threshold_sdk@v0.1.0/threshold_core@v0.1.0. These are v0.1.0 public modules with no track record. The DKG ceremony correctness (especiallya0leader secret re-use across reshares to pin the group key) should be reviewed by a cryptographer. -
Attested admission — Followers are admitted via mutually attested handshake over TLS. The PCR check (
nitriding.ArePCRsIdenticalForKeys) must be reviewed carefully — a lenient PCR check would allow a compromised node to join the quorum. -
Share persistence atomicity — Leader writes
{leader_secret, share}atomically to SSM. If the write fails between the two fields, the node has an inconsistent state. Verify error handling covers this. -
Group restart on reshare — The lifecycle triggers
RequestUpstreamRestart(SIGTERM → 10s → re-fork) on new share receipt. Verify that in-flight signing operations are drained before restart, and that a failed restart doesn't leave the signer in a non-responsive state while still holding a valid share. -
Dev mode kernel-cmdline —
devcmdline.gomaps kernel-cmdline tokens to env vars in dev mode. Verify this path is entirely gated on the measuredENCLAVE_DEVflag and cannot be activated in production images. -
No quorum = no signing — If the threshold is t-of-n and fewer than t nodes are live, signing halts. This should be documented as an availability tradeoff — especially relevant if funds are gated on enclave signing.
This needs expert cryptographic review before merge.
|
This PR has been open for 5+ weeks without review. @aruokhai this is a security-critical cryptographic feature (threshold signing DKG). Is there a reviewer assigned? |
|
This PR has been open for 41 days without a review decision. @aruokhai is anyone looking at this? |
|
This PR has been open for 44+ days without review. @aruokhai is anyone looking at this? |
|
This PR has been open for weeks without review. @aruokhai is anyone looking at this? |
|
This PR has been open for 44+ days without review. @aruokhai is anyone looking at this? |
|
This PR has been open for 49 days without a review decision. @aruokhai is anyone looking at this? The Leader/Follower Secret Sharing design looks significant — would benefit from a dedicated reviewer. |
|
This PR (Leader/Follower Static Secret Sharing, ~55 days open with no review) implements a new threshold-cryptography scaling model for the enclave. Given it touches key admission, ceremony coordination, and secret distribution, it needs a thorough security review by someone with experience in threshold schemes before merging. @aruokhai — is this still actively maintained? A review assignment would help move it forward. |
|
This PR has been open since July 1 without review (nearly 2 months). @aruokhai is anyone looking at the Leader/Follower Static Secret Sharing? |
|
This PR (Leader/Follower Static Secret Sharing) has been open for 66 days without review. @aruokhai is this still being pursued? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Arkana review — #139 (sha 5eaa2de)
Leader/Follower Static Secret Sharing (threshold scaling via ChillDKG/FROST)
Assessment: the architecture is well-documented and the mutual-attestation design is principled; the cryptographic implementation and ceremony protocol need expert review.
What changed
Introduces a threshold-scaling model where signing secrets are split across a leader and N followers using ChillDKG/FROST. Each node holds only a share; a quorum is required to sign. Membership changes trigger a reshare (proactive security). The leader holds the original secret (sealed at rest); followers only hold their share.
Architecture observations
- Mutual attestation: the handshake requires both parties to verify each other's PCR0 against their own — a rogue leader cannot harvest join attempts. The nonce is single-use with a 60s TTL. This is the correct design.
- Sealed relay channel: the leader's response is encrypted to the follower's ephemeral box key inside the signed attestation. This is correct — the relay payload is confidential to the attesting follower.
- Proactive resharing: every membership change reshares all shares, making captured old shares useless. Good.
- Image-scoped identity: only image/kernel/application PCRs are compared, not host-specific values. This allows different-host enclaves to admit each other as long as they run the same image. Correct for the stated use case.
- Follower state is ephemeral: membership is not persisted; a rebooted follower re-runs admission and the next reshare folds it back in. Shares are KMS-persisted per node.
Concerns for human reviewer
- The leader holds the original secret: this is explicitly stated ("the leader secret, sealed at rest, never exported"). This means the leader is a higher-value target than any follower — if the leader's enclave is compromised and the secret is extracted from KMS (which would require breaking PCR attestation), the group key is exposed. This is a known trade-off; confirm it's acceptable for the use case.
- Reshare ceremony correctness: the SCALING-ARCHITECTURE.md describes the ceremony but the actual cryptographic implementation (via
ArkLabsHQ/threshold-magic) is external to this repo. The PR should link to the version pinned, and that library's ceremony correctness should be reviewed separately. - Roster injection: the leader broadcasts the full member roster in each ceremony start command. A compromised leader could present different rosters to different followers. The claim is "the ceremony's final round has every participant sign the same session transcript (which includes the roster), so if any two nodes were fed different rosters the ceremony fails." Verify this claim holds in the FROST implementation used — specifically that the transcript commits to the full roster in a way that makes forking detectable.
- CI uses QEMU: the scaling test runs in a QEMU-emulated enclave. Confirm that the attestation verification path tested in CI (with
ENCLAVE_DEV=true) exercises the same code paths as production, modulo the COSE/cert verification skip. - `kind: signing` secret delivery to the application: the architecture doc says shares arrive asynchronously on a
Keys()channel. How does the application consume this? If the application starts before the first ceremony completes, does it block, fail, or operate in a degraded mode?
Scope question
This PR introduces substantial new infrastructure. Is it gated behind a feature flag in production? The kind: signing tag on individual secrets suggests it's opt-in per secret — confirm that a deployment with no kind: signing secrets is entirely unaffected.
This work is impressive in scope and the architecture documents are thorough. The cryptographic ceremony correctness and the external threshold-magic dependency need expert review before production enablement.
|
This PR has been open for 79+ days without a review decision. @aruokhai — need any help addressing the feedback or rebasing? |
|
This PR has been open since July 1 (nearly 3 months) with no review decision. @aruokhai is there anything blocking a review? This is a security-critical cryptographic feature (Leader/Follower Static Secret Sharing / DKG) that should have expert eyes on it. |
2 participants
Leader/Follower threshold-shared static secrets
Adds a threshold-cryptography scaling model for enclave apps. A secret tagged
kind: signingis shared amongst enclaves: each node keeps only a share, a quorum is required to sign, and thegroup public key stays pinned to the original secret as nodes join or leave. Ordinary
secrets and single-node deployments are unchanged (opt-in).
Full design:
SCALING-ARCHITECTURE.md.What's in scope
ScalingEntity(runtime/scaling.go), built inRuntime.Initonly whenENCLAVE_SCALING_ROLEis set (nilotherwise). Leader runssdk.NewLeader+ the gRPCrelay; follower runs
sdk.NewJoiningFollower. Both hold a KMS-persisted secp256k1 hostidentity key.
DkgStore(runtime/chilldkg.go) adaptssdk.Storeontoencrypted
Storage;GetLeaderSecretreturns the leader's original secret (a0) soevery ceremony reconstructs the same group key.
port:
GET /enclave/handshake/noncethenPOST /enclave/handshake/admit. The leaderverifies the COSE signature and PCRs 0/1/2 (
nitriding.ArePCRsIdenticalForKeys), thenseals
{leaderPub, relayPort}to the follower's ephemeral key. The admitted host keybacks
sdk.WithAuthorize, so the relay rejects unadmitted keys.a0; all shares re-randomize, thegroup key holds. Results arrive on
Keys(); a listener persists the sealed envelope,sets the secret's env var, and triggers a graceful upstream restart
(
RequestUpstreamRestart→ SIGTERM/10s → re-fork) so the app adopts the new share.(
sharedSecret{leader_secret, share}); leader writes both atomically, followers onlyshare.runtime/devcmdline.gomaps whitelistedenclavecfg.*kernel-cmdlinetokens to env vars (no-op unless the measured
ENCLAVE_DEVflag is set), lettingidentical-PCR QEMU enclaves take distinct roles.
Testing
.github/workflows/scaling-test.yml,test/run-scaling.sh,test/scaling-test.sh. Boots 1 leader + 2 followers of the same EIF on QEMU/KVM andasserts: nodes healthy/role-correct, handshake live, every node holds a share (not
the leader secret), the 2nd join re-randomizes the 1st follower's share, all shares
distinct valid scalars.
Notes
threshold_sdk@v0.1.0/threshold_core@v0.1.0as public versioned modules.