mn-python-sdk provides the Python gRPC client and workflow-bundle helpers used
by the CLI, API, and Python-defined workflows.
mn_sdk.payload_contract validates blueprint-owned skill and agent source
packages or wheels under payloads/skills and payloads/agents. It also
validates GGUF, Safetensors, and DDUF declarations below payloads/models,
including auxiliary files and SHA-256 digests. Payload packages override exact
matching GAR dependencies.
iter_bundle_assets and stage_payload_assets stream large files into
content-addressed storage; model files are never eagerly buffered into the
submission payload. package_payload_models prepares physical model payloads
with Docker Model Runner. Submission preparation renders a bundled agent index
and injects payload Python packages into HostLocal environments. When a
HostLocal upload source imports mn_sdk, submission preparation stages the
SDK package beside that source so the isolated worker receives the same client
contract used to prepare its job. Compiled bundles marked
python_source_mode=false keep their generated dependency-light source
fallback and are not expanded with the full SDK at submission time.
mn_sdk.llm provides LLMClient.from_env(), text and strict JSON completions,
retries, deterministic fallbacks, and token usage accounting. Strict JSON calls
use the configured retry allowance for empty, incomplete, or invalid structured
responses as well as transport failures; a retry sends only a bounded corrective
instruction and never reflects the invalid response back into the prompt. The lower-level
mn_sdk.model_access.runtime wrapper is shared by LLM, RAG embedding, and OCR
calls.
Prepared blueprint submissions stage run-store writes in shared storage and
register a run-scoped copy-back to the configured local MN_RUNS_ROOT.
Terminal copy-back observes the authoritative source tree until it is stable
and recopies late-arriving files, preventing a completed local run directory
from retaining only a partial artifact set.
Local-development dependency staging also carries manifest-declared versions into copied setuptools-scm package sources. This keeps source-built sibling skills compatible when the staged Docker context does not include Git metadata.
Managed Docker Model Runner models are selected and prepared on first use.
Logical default prefers Nemotron 3 on a healthy compatible cluster node and
otherwise uses catalog fallback Gemma 4. Non-LLM consumers pass a complete
model specification to the generic wrapper; for example, the RAG and OCR
skills own their concrete model, backend, context, and hardware requirements.
Hardware preflight also accepts a healthy federated Core that can own and
locally schedule the forwarded Job; that does not make it directly schedulable
by the submitting Core.
Those models are not SDK catalog defaults or blueprint declarations. Successful
bindings are cached per worker process; the native runtime service supplies
install single-flight and LiteLLM routing. External OpenAI-compatible, LiteLLM,
and Ollama endpoints bypass installation.
After both DMR preparation and gateway routing succeed, the native runtime
service idempotently registers the selected model in
$MN_HOME/models/registry.json. This applies to a newly installed artifact and
to one that was already present in Docker Model Runner, so successful lazy
first use appears as managed in mn model list.
For a local Compose runtime, native resource advertisement may use the
MN_NATIVE_SDK_GRPC_TARGET service address when no separate advertise host is
configured. A cluster advertise host still takes precedence when present.
Blueprint actor configuration preserves docker_model_runner as the provider
for managed models. LiteLLM is the node-local gateway transport after model
preparation, not a replacement provider contract that bypasses preparation.
The node-local LiteLLM gateway applies one shared FIFO admission queue across
parallel workers and model routes. It defaults to one active request and 64
waiters, so excess calls wait instead of starting enough local decoders to
starve Core. Chat/completion calls and embeddings use separate bounded FIFO
lanes so long local decodes cannot starve short RAG requests. Configure them
with MN_LITELLM_MAX_CONCURRENT_REQUESTS,
MN_LITELLM_MAX_CONCURRENT_EMBEDDINGS, MN_LITELLM_MAX_QUEUED_REQUESTS,
MN_LITELLM_QUEUE_TIMEOUT_SECONDS, and MN_LITELLM_MAX_SLOT_SECONDS. Queue transitions are emitted as structured
runtime_llm_request_* log events and responses report queue wait time in the
x-mn-llm-queue-wait-ms header.
Managed model selection and preparation use the dedicated
MN_RUNTIME_MODEL_PREPARE_TIMEOUT_SECONDS deadline (20 minutes by default), so
first-use cluster-resource discovery, model routing, and downloads are not cut
off by the general 10-second gRPC client deadline.
DockerWorker model selection prefers the worker's actual execution node before
considering installation reuse or capacity on another node. This keeps a
node-local workflow on its selected machine and returns the reachable
mn-litellm-proxy address. If that node cannot satisfy the model requirements,
selection may use another compatible cluster node and its advertised gateway
address. Core supplies the actual worker placement as MN_EXECUTION_NODE; that
value takes precedence over the submission-time MN_SELECTED_RUNTIME_NODE.
When Docker reports that node on its bridge address while Core uses the LAN
address, their shared node name is treated as the same local node.
Submission preparation also injects the Docker-reachable
MN_RUNTIME_MODEL_CONTROL_TARGET (host.docker.internal by default) into
managed DockerWorker nodes so their first-use preparation RPC reaches Core
and node-local native SDK through the host bridge, instead of container
loopback or the host's advertised LAN address.
mn.workflow/v1 source-form manifests (kind: WorkflowSource) declare direct
DAG dependencies with needs and select Python behavior modules with
run.handler:
{
"id": "research",
"needs": ["intake"],
"run": {
"handler": "my_blueprint.steps.research",
"with": {"operation": "company_identity"}
}
}The module defines run(); the manifest does not need a :run suffix. During
expansion, the standard blueprint profile configures each handler-backed
worker to run python3 -m mn_sdk.step_runtime. The SDK entrypoint invokes only
the scheduler-selected handler and passes a StepContext containing the step
id, run id, attempt metadata, incoming message, and embedded config.
A logical step can instead reference a Python StepSpec. Registry entries own
immutable agent handlers and parameters, while the step module owns its input
contract, output contract, and internal collaboration graph:
{
"agents": {
"registry": {
"extractor": {"handler": "my_blueprint.agents.extractor"},
"normalizer": {"handler": "my_blueprint.agents.normalizer"}
}
},
"workflow": {
"steps": [{
"id": "prepare",
"needs": [],
"run": {"definition": "steps.prepare"}
}]
}
}from mn_sdk.step_graph import (
InputSpec,
OutputSpec,
StepSpec,
agent,
flow_output,
run_input,
sequence,
)
STEP = StepSpec(
input=InputSpec(fields={"document_folder": run_input("document_folder")}),
flow=sequence(
agent("extractor", as_="extract"),
agent("normalizer", as_="normalize"),
),
output=OutputSpec(fields={"company_evidence": flow_output()}),
)The compiler expands each logical step into a start boundary, its internal
agent graph, and an end boundary. It supports sequence, all-required
parallel, choice with a
default, fallback, and bounded_loop. Workflow edges connect only a previous
step's end boundary to the next step's start boundary. Agent handlers use
receive_input(context) and send_output(...); Redis routing, retries,
fan-out, and fan-in remain outside agent code.
Blueprint runtime contexts keep persisted metadata across retries. If a saved run or output path is not mounted in the current runner, the SDK retains the currently resolved path instead; this keeps durable state portable across DockerWorker, OpenShell, and host execution boundaries.
Install locally and run tests:
python3.11 -m venv .venv
. .venv/bin/activate
.venv/bin/python -m pip install -e ".[dev]"
.venv/bin/python -m pytest -q
.venv/bin/python -m ruff check .mn_sdk.public is the small, library-friendly surface for starting and
monitoring jobs. When the script runs on the MirrorNeuron host, no connection
arguments are needed: it reads the active gRPC endpoint and connection token
through the same runtime configuration used by the CLI, including
$MN_HOME/runtime-endpoints.json and $MN_HOME/grpc_auth.token.
from mn_sdk.public import MirrorNeuron
with MirrorNeuron() as mn:
run = mn.launch(
"/path/to/bundle",
inputs={"query": "example"},
)
for event in run.events():
print(event)
final = run.wait(timeout=600)
another = mn.start(run.job_id, inputs={"query": "another"})launch() accepts a bundle directory containing manifest.json, creates one
durable job definition, and starts its first run. start() reuses that durable
job for another run. Events and status records are returned as raw JSON
objects. A failed or cancelled run is returned by wait() as a terminal
record; only a caller deadline raises RunTimeoutError.
Explicit connection settings remain available for remote callers and override local discovery. Passing an empty connection token explicitly disables token authentication.
with MirrorNeuron(
target="core.example:55051",
connection_token="...",
) as mn:
run = mn.launch("/path/to/bundle")Async consumers use the matching non-blocking façade:
from mn_sdk.public import AsyncMirrorNeuron
async with AsyncMirrorNeuron() as mn:
run = await mn.launch("/path/to/bundle", inputs={"query": "example"})
async for event in run.events():
print(event)
final = await run.wait(timeout=600)The public façade intentionally does not expose job administration, schedules,
deployments, cluster mutation, or terminal UI behavior. Those capabilities
remain on the existing mn_sdk.Client, CLI, and API surfaces.
See the detailed public Python SDK guide for
configuration precedence, the complete method reference, lifecycle and error
semantics, async and library-integration patterns, and operational notes. A
runnable example is available at
examples/public_sdk_launch.py.
Minimal client example:
from mn_sdk import Client
client = Client(target="localhost:55051")
print(client.list_jobs(limit=5))The sole v1 job contract retains configuration or data across executions:
import json
from mn_sdk import Client
client = Client(target="localhost:55051")
job = json.loads(client.create_job(manifest_json, payloads))
first = json.loads(client.start_run(job["job_id"], inputs={"source": "manual"}))
second = json.loads(client.start_run(job["job_id"], inputs={"source": "scheduled"}))
assert first["run_id"] != second["run_id"]job_id owns the durable definition and $MN_HOME/job-data/<job-id>.
run_id owns one execution and all control/observability calls. Retries keep
their run ID and use attempt_id. RuntimeService exposes create, get, list,
update, archive, reset-data, delete, start/list-run, run-control, run-delete,
and job-schedule adapters for mirrorneuron.job.v1.JobService.
update_job(..., manifest_json=..., payloads=...) atomically replaces
an inactive job's executable bundle. The graph and blueprint identities must
match; job data, schedules, and prior run records are preserved.
Deleting or replacing a definition retires its SDK-prepared shared-storage and
DockerWorker resources from Core's metadata-only resource descriptor.
Transport-size exhaustion is reported separately from runtime resource
pressure, so an oversized server response is not presented as a busy cluster.
A source or executable manifest can opt one stable Job into a definition-level response service without adding a workflow node or service port:
{
"response_service": {
"enabled": true
}
}The declaration accepts no commands, handlers, nodes, ports, or other fields.
RuntimeService.query_job_response() sends one bounded question and the
sanitized mn.mcp.job_context.v1 projection through
JobService.QueryJobResponse. It does not start a Run. The optional
conversation ID is a UUID and the optional request ID is an idempotency key of
at most 128 characters.
Running jobs can expose narrowly scoped commands through
contracts.live_inputs. Callers submit only the public input ID and payload;
the manifest supplies the entrypoint and message type:
accepted = json.loads(
client.send_run_input(
first["run_id"],
"steer_monitoring",
{"instruction": "Watch the loading dock", "analyze_now": True},
idempotency_key="operator-command-001",
)
)Service agents may request a collision-free runtime port with
resources.ports[].port: "auto". Core replaces it with an integer in the
scheduler allocation and supplies MN_PORT_<LABEL> to the worker. A service
declaration should publish that same value through a template such as
"${env.MN_PORT_MCP_COLLABORATION}".
Source manifests can start independent supervised service nodes without
replacing workflow roots by listing their IDs in
agents.auxiliary_entrypoints. The compiler places these service entrypoints
before workflow roots so supervision can begin before short-lived work nodes.
The manifest compiler validates each live-input ID, object schema, declared entrypoint, and permitted message route. Shared blueprint-support dashboards derive their controls from those declarations and can optionally publish a credential-free HLS preview through Core's supervised media-relay node.
Runtime payloads must read MN_RUN_ID for execution identity. The SDK no
longer treats MN_JOB_ID as a fallback run ID. MN_JOB_ID, MN_RUN_ID,
MN_ATTEMPT_ID, and MN_JOB_DATA_DIR therefore have distinct meanings.
prepare_job_submission(..., env=...) accepts an explicit environment mapping
for local DockerWorker preparation. This lets adapters enable diagnostic Docker
build output or provide runtime-local settings without changing process-global
environment variables.
Prepared submissions preserve durable-job, run, and attempt identity as
separate fields. Run stores and staged artifacts are keyed by run_id; durable
job data is mounted by Core and is never synthesized from a caller-provided
host path.
prepare_manifest_submission() is the shared CLI/API adapter boundary. It
returns the executable manifest, resolved blueprint config, and effective
runtime environment; prepare_manifest_for_submission() remains the
manifest-only convenience wrapper over that same implementation.
write_blueprint_job_mapping() persists the shared job/run identity mapping
and a sanitized public workflow manifest so monitors never expose lowered
runtime control nodes as user-facing steps.
The caller generates job_id before preparation. Shared storage and
DockerWorker services use a job-scoped definition-revision ID, never a
run_id, so ordinary starts and scheduled dispatches reuse the stored
definition resources.
When that mapping contains MN_SELECTED_RUNTIME_NODE, DockerWorker preparation
treats it as the workflow's hard placement target. This is required for
v1 source manifests because DockerWorker nodes are generated after the CLI has
made the hardware-fitness decision. The SDK resolves the selected node from the
injected/current cluster reports and uses its native SDK client; it does not
fall back to a local Docker build when the selected node is remote.
The shared workflow resolver also applies a hard node.name constraint to
every executor and to every source, sink, join, router, aggregator, or other
control node generated during lowering. Nodes with a divergent or read-only
coordination store are not placement candidates.
When the mapping also enables MN_DEBUG or MN_BLUEPRINT_DEBUG, that intent is
carried through remote native-SDK preparation. The returned DockerWorker service
record includes the build action, image, command, context digest, and complete
captured output so the calling CLI can show remote Docker build diagnostics.
Small DockerWorker build contexts are sent directly to the selected node's
native SDK so a newly joined cluster does not block behind an unrelated
Syncthing backlog. Only the build-context subtree is included in that native
request. The default bound is 3 MiB, below the standard gRPC message ceiling.
Larger contexts continue to use digest-verified shared-storage staging. Set
MN_DOCKER_WORKER_INLINE_PAYLOAD_MAX_BYTES or
MN_DOCKER_WORKER_INLINE_CONTEXT_MAX_BYTES to tune the bounds; set either to
0 to force shared-storage staging.
The boundary is service-free in unit tests: inject cluster_client,
native_client_factory, and command_runner into
prepare_docker_worker_compose_services. The focused regression is:
../mn-system-tests/.venv/bin/python -m pytest -q \
tests/test_native_resources.py \
-k "selected_runtime_node or cuda_docker_worker"Blueprints use apiVersion: mn.workflow/v1 with kind: WorkflowSource for a compact,
CSS-like manifest.json that declares intent and overrides while SDK profiles
provide common defaults. Generate the executable runtime manifest with:
mn-manifest-converter expand manifest.json --output build/manifest.executable.json
mn-manifest-converter check manifest.json --against build/manifest.executable.jsonPre-submission validation accepts input_validation either as a source-level
section or under the source manifest's manifest section. Both forms are
validated before source expansion and runtime preparation. In local-development
mode, command validators can import only the local skill sources declared by
that manifest.
The CLI/API expand source manifests automatically before validation and
submission. Expansion returns an executable mn.workflow/v1 manifest with
kind: Workflow; submission preparation then lowers it to Core's runtime
topology, including graph_id. The retired mn.workflow.source/v2 and
mn.workflow/v2 values are rejected.
For source-form v1 blueprints, config.manifest_defaults can expose authoritative
manifest descriptors through resolved runtime configuration without copying
them into config/default.json. A dotted string keeps the same path; a mapping
projects it to another config path:
{
"config": {
"manifest_defaults": [
"llm",
{"from": "requirements", "to": "resources"}
]
}
}Manifest values are merged first, followed by the default config file and the
invocation overlay. Both manifest compilation and load_runtime_config() use
this order.
workflow.requires and workflow.provides compile into runtime DAG edges.
Declare an explicit workflow.edges list when an edge needs a custom event or
otherwise cannot be inferred from a provided capability. A step can declare a
runtime trigger at trigger_rule (or control.trigger_rule): all_success,
all_done, one_success, one_done, one_failed,
none_failed_min_one_success, or quorum_success with a positive quorum.
The generated manifest places these under flow.steps and flow.graph.edges,
which are consumed by the Core workflow ledger.
Configuration is loaded by mn_sdk.config in this order:
real environment variables
> .env.${MN_ENV}
> .env
> built-in safe defaults
MN_ENV defaults to dev when unset. MN_ENV=development loads .env.dev;
MN_ENV=test loads .env.test; MN_ENV=prod or MN_ENV=production loads
.env.prod when present. Production does not require any .env file.
Development example:
export MN_ENV=dev
cp .env.example .env.dev
mn-cli ...Test example:
export MN_ENV=test
mn-cli ...Production example:
export MN_ENV=production
export MN_HOME=/var/lib/mirrorneuron
export MN_LOG_LEVEL=info
export MN_API_HOST=0.0.0.0
export MN_API_PORT=8080
mn-api ...Operator-added models are stored atomically in
$MN_HOME/models/registry.json. The SDK registry accepts normalized Docker
Model Runner records and the canonical provider JSON represented by
mn-docs/examples/openai-compatible-model-proxy.json. Provider records retain
only environment-variable references such as os.environ/OPENAI_API_KEY; the
resolved secret value is never persisted.
Registry entries are projected into model resolution and the managed LiteLLM
gateway. Legacy models/proxies.json and manually registered
model-remotes.json entries are not imported or projected. Remote records
owned by run_cluster_model_monitor remain separate and active.
An operator-selected registry default is authoritative and is not silently
replaced by a built-in fallback when its upstream is unavailable.
OpenAI-compatible definitions may retain an upstream model name such as
muse-glimmer-30b; the registry adds LiteLLM's openai/ transport prefix
when it projects that model through the managed gateway. A custom endpoint
without apiKeyEnv receives LiteLLM's non-secret not-needed placeholder.
The SDK uses the packaged mn_sdk/model_catalog.json as its baseline catalog.
If present, $MN_HOME/models/catalog.json is loaded next; $MN_HOME defaults
to ~/.mn. Entries are deep-merged by model ID, so an external entry can
override selected fields while unmentioned built-in models remain available.
Set MN_MODEL_CATALOG_PATH to load a final, highest-priority catalog file.
The file may be a model list, an object with a models list, or an object keyed
by model ID. Paths support ~, $MN_HOME, and normal environment-variable
expansion.
For example, this changes the bundled Gemma model endpoint and adds a new catalog entry without copying the entire packaged catalog:
mkdir -p "$MN_HOME/models"
cat > "$MN_HOME/models/catalog.json" <<'JSON'
{
"models": [
{
"id": "gemma4:e2b",
"model": "local/gemma4:E2B",
"requirements": {"min_vram_gb": 4}
},
{
"id": "my-local-model",
"model": "local/my-model",
"aliases": ["my-model"]
}
]
}
JSONCatalog precedence is:
- Packaged
mn_sdk/model_catalog.json. $MN_HOME/models/catalog.json, when present.MN_MODEL_CATALOG_PATH, when configured.
Matching entries are merged by id. Nested objects are merged recursively;
scalar values and lists from the higher-priority catalog replace lower-priority
values. A malformed existing catalog raises a validation error rather than
being silently ignored.
Do not commit real .env files. Use .env.example for placeholders only, and
put secrets in real environment variables or token files.
- A running MirrorNeuron core is required for live client calls.
- Constructor arguments take precedence over environment variables.
- Generated protocol modules are included with the package.
Client.start_operation(kind, options) starts a server-defined durable bulk
operation and returns its JSON snapshot. Use get_operation(operation_id) to
read the latest state and stream_operation_events(operation_id, after_sequence=..., follow=True) to replay/continue progress after a detach.
The operation kinds are cancel_all_jobs, clear_jobs, reconcile_node, and
drain_node; target selection and concurrency remain Core-owned.
cancellation_pending is an accepted cancellation result: the Core has fenced
the old owner and queued cleanup for its rejoin. It is not a per-item failure.