Build a reproducible acceptance harness for confidential workflows across components

Problem and scope

Component tests do not establish that a composed model, agent, tool and delegated peer preserve a selected confidentiality policy.

Provide a reusable acceptance harness with replaceable model, tool, transport, attestation and deployment adapters. Start with a deterministic synthetic workflow; label fresh hardware evidence separately.

Acceptance criteria

Define assets, principals, permitted recipients, trusted components and the exact claim evaluated by each run.
Join key provisioning, inference, a mediated tool call, a delegated peer call and response verification using a single bound transaction context.
Capture independent observations at release, receipt and execution boundaries; keep authorization, delivery, installation and execution outcomes separate.
Cover workload/key/configuration substitution, software downgrade, replay, missing evidence, bypass egress, forbidden disclosure, revocation and timeout with unknown outcome.
Pair expected refusals with valid controls and show that removing each tested gate causes its intended test to fail.
Produce machine-readable results with source revisions, dependency versions, policy/trust inputs, commands and explicit synthetic/replayed/live evidence classification.
Reproduce a live run across independently operated peers satisfying the selected policy. Sanitized public evidence must not expose protected data or device identifiers.

Dependencies and closure

Consume the response-binding, confinement, measured-identity, lifecycle, protected-execution and disclosure workstreams linked from the parent. agentrust-io/ca2a#47 remains the owner of cA2A hardware conformance and release-assurance decisions. A synthetic harness can complete its software milestone without closing live-hardware acceptance or promoting a component's release claim.

Published integration examples must use released packages, following this repository's contribution rules. Before required APIs are released, keep protocol development in the owning core repositories and mark the integrated milestone as waiting on those releases.

I would like to take the bounded synthetic/software milestone for #199.

I checked the integrations contribution rule before starting implementation. The current published releases are cMCP 0.5.0 and cA2A 0.2.0; the newer exact-output disclosure work and recent response-binding work landed after those releases. Because this repository requires published integrations to run against released packages, I do not want to build the public harness against main/unreleased APIs and then call it reproducible.

Proposed shape:

1. One run manifest, one explicit claim

Each run begins with a machine-readable manifest that states:

run ID / transaction context
protected assets and sensitivity
workload principal, represented user/delegator where present, delegated peer
permitted recipients / forbidden recipients
policy version and trust inputs
component package versions and source revisions
evidence classification per component: synthetic, replayed, hosted/live, hardware
exact claim under evaluation, e.g. “asset X may be processed by workload A, delegated to peer B within scope S, and released only to recipient R under policy P”

The result must never silently upgrade a synthetic component because another component used live evidence.

2. Replaceable adapters, fixed harness semantics

Keep the harness neutral and make components adapters:

key/provisioning adapter
model/inference adapter
mediated-tool adapter
delegated-peer / cA2A adapter
response-verification adapter
disclosure/release adapter
attestation/deployment adapter
independent observation adapters

Adapters return typed observations. They do not decide the overall verdict themselves.

3. Separate outcome dimensions

Do not collapse “authorized” into “happened.” Record at least:

authorization
delivery
receipt
installation/admission
execution
response verification
release/disclosure

Each can be established / contradicted / unavailable (or an equivalent closed vocabulary). A timeout after dispatch should remain unknown, not become success or failure by inference.

4. First deterministic synthetic workflow

Positive control:

key provisioning -> deterministic model/inference step -> mediated cMCP tool call -> scoped delegation to cA2A peer -> peer response -> response verification -> permitted release

The same transaction context must be bound across the artifacts used by the run; a post-hoc log correlation alone should not satisfy that criterion.

Negative cases:

workload substitution
key substitution
effective-config/policy substitution
software downgrade
replay
missing evidence
bypass egress
forbidden disclosure
revocation after authorization / before use
timeout after dispatch with unknown outcome

Each refusal gets a paired valid control.

5. Mutation proof for every claimed gate

For each negative case, deliberately remove or weaken the relevant gate and require the corresponding test to turn red. This prevents a “green” harness whose expected refusal is actually caused by some unrelated earlier check.

6. Evidence bundle

Emit a machine-readable result containing:

scenario ID
run/transaction ID
package versions and source revisions
commands
policy/trust input digests
adapter/evidence classifications
per-boundary observations
expected vs observed outcome
mutation-control result
explicit limitations

No protected payloads, device IDs, secrets or low-entropy confidential digests in the public evidence.

Release gate

I suggest we keep protocol/API development in the owning core repos until the required response-binding/disclosure surfaces are published, as CONTRIBUTING requires. Once compatible releases exist, the first integrations PR can pin those exact package versions and deliver the deterministic software milestone. The live independently-operated-peer/hardware milestone remains separate and should not be implied by the synthetic pass.

I can prepare the harness layout and test matrix against this contract now, but would avoid submitting code that imports unreleased APIs.

@altrudev

@altrudev please go ahead with the harness layout and test matrix for the synthetic/software milestone. That matches the scope here.

Keep protocol/API changes in the owning core repos, and pin released package versions when wiring up the integration. Mark any blocked cases as waiting on those releases. A synthetic pass completes only the software milestone; live peer and hardware acceptance remain open.

I reviewed the synthetic/software milestone again as an evidence-boundary problem before turning it into code. I think the harness should be split into a small fixed core plus replaceable adapters, with the test matrix proving each gate independently rather than treating one end-to-end green run as sufficient.

Harness layout

confidential-workflow-harness/
  README.md
  manifest.schema.json
  result.schema.json
  scenarios/
    positive-control.yaml
    workload-substitution.yaml
    key-substitution.yaml
    policy-substitution.yaml
    software-downgrade.yaml
    replay.yaml
    missing-evidence.yaml
    bypass-egress.yaml
    forbidden-disclosure.yaml
    revocation-before-use.yaml
    timeout-after-dispatch.yaml
  harness/
    model.py
    types.py
    runner.py
    verdict.py
    evidence.py
    adapters/
      provisioning.py
      inference.py
      mediated_tool.py
      delegated_peer.py
      response_verifier.py
      release.py
      attestation.py
      observation.py
  tests/
    test_positive_control.py
    test_negative_matrix.py
    test_mutation_proofs.py
    test_evidence_bundle.py
    test_unknown_outcomes.py

The harness core owns transaction binding, scenario sequencing, evidence classification and verdict semantics. Adapters emit observations only; they cannot upgrade the overall verdict.

Closed outcome vocabulary

For each boundary, record one of:

established
contradicted
unavailable
not_applicable

and keep these dimensions independent:

authorization
delivery
receipt
installation/admission
execution
response verification
release/disclosure

A timeout after dispatch therefore yields, for example, delivery=established and execution=unavailable; the harness must not infer execution from dispatch.

Binding rule

Each artifact used to establish the positive claim must carry or be cryptographically/structurally bound to the same transaction context. Timestamp proximity, log adjacency and request correlation are not sufficient substitutes.

A run manifest should minimally bind:

run/transaction ID
protected asset identifier (non-secret)
workload principal and represented/delegating principal
delegated peer
permitted and forbidden recipients
policy version/digest
trust-input digest
package versions/source revisions
evidence classification per adapter
exact claim under test

Test matrix

Scenario	Expected boundary	Expected result	Mutation proof
positive control	all required software boundaries	established	n/a
workload substitution	admission/execution	contradicted/refused	disable workload identity gate -> scenario must turn green
key substitution	authorization/verification	contradicted/refused	trust substituted key -> scenario must turn green
policy/config substitution	authorization/release	contradicted/refused	bypass policy binding -> scenario must turn green
software downgrade	admission	contradicted/refused	remove version floor -> scenario must turn green
replay	authorization/receipt	contradicted/refused	remove nonce/replay gate -> scenario must turn green
missing evidence	affected boundary only	unavailable	synthetic evidence insertion -> expected control becomes established
bypass egress	release	contradicted/refused	bypass release gate -> scenario must turn green
forbidden disclosure	release	contradicted/refused	authorize recipient -> paired control becomes established
revocation after authorization / before use	execution/release	contradicted/refused	suppress revocation check -> scenario must turn green
timeout after dispatch	execution	unavailable	deterministic completion control -> established

For every negative case, the paired valid control should differ only in the property under test.

Evidence bundle

Each run should emit one machine-readable result containing:

scenario ID and transaction ID
exact claim
package versions and source revisions
commands/entry point
policy and trust-input digests
adapter type + evidence classification (synthetic, replayed, hosted/live, hardware)
ordered observations with causal source/provenance
per-boundary outcomes
expected vs observed result
mutation-proof result
blocked/waiting-on-release markers
explicit limitations

No protected payloads, secrets, device identifiers or low-entropy digests of protected material should enter the public evidence bundle.

Evidence-integrity checks

Two additional checks seem important for this milestone:

branch preservation: retries/alternate paths remain separate lineages rather than being flattened into one successful timeline;
post-action contamination: later retry/recovery/reconciliation evidence cannot silently replace the evidence that existed at the earlier decision boundary.

These are harness-level properties and do not require protocol/API changes.

Release boundary

I will keep the initial implementation strictly on published package surfaces. Where a scenario depends on unreleased response-binding or disclosure APIs, the test should be present but explicitly marked waiting_on_release; it must not be simulated in a way that produces a false pass.

The software milestone can then be called complete only when:

the deterministic positive control passes on released packages,
every available negative case and its mutation proof behaves as expected,
blocked cases remain explicitly blocked rather than inferred,
the evidence bundle validates against schema, and
no result implies live-peer or hardware acceptance.

If this shape looks right, the next step is to implement the fixed core/schema + deterministic adapters, then wire released cMCP/cA2A surfaces as they are available.

@altrudev

@altrudev, go ahead with a focused PR for the core, schemas and deterministic software adapters.

Keep the valid comparison cases, but also weaken each safety check and show that its test fails. Making an input valid is a separate control.

If a required case is waiting on a release, keep the overall software milestone incomplete and report which parts passed. Pin released packages; live-peer and hardware acceptance remain separate.

Navigation Menu

Repository navigation

Build a reproducible acceptance harness for confidential workflows across components #199

Problem and scope

Acceptance criteria

Dependencies and closure

1. One run manifest, one explicit claim

2. Replaceable adapters, fixed harness semantics

3. Separate outcome dimensions

4. First deterministic synthetic workflow

5. Mutation proof for every claimed gate

6. Evidence bundle

Release gate

Harness layout

Closed outcome vocabulary

Binding rule

Test matrix

Evidence bundle

Evidence-integrity checks

Release boundary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Build a reproducible acceptance harness for confidential workflows across components #199

Description

Problem and scope

Acceptance criteria

Dependencies and closure

Activity

altrudev commented on Sep 19, 2026

1. One run manifest, one explicit claim

2. Replaceable adapters, fixed harness semantics

3. Separate outcome dimensions

4. First deterministic synthetic workflow

5. Mutation proof for every claimed gate

6. Evidence bundle

Release gate

imran-siddique commented on Sep 20, 2026

altrudev commented on Sep 21, 2026

Harness layout

Closed outcome vocabulary

Binding rule

Test matrix

Evidence bundle

Evidence-integrity checks

Release boundary

imran-siddique commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions