Repository navigation
Package/expose available configs and workflows #737
Description
Hello!
I wanted to ask if it would be possible to install only the configs via a package?
Our thought is that maybe then we can install the config package in NICOS without all the other dependencies. Then it would be easier to make sure that message formats are as expected on both the nicos and esslivedata side.
Otherwise we had the suggestion of you publishing available configs/workflows in kafka, so whenever the service is running, NICOS will always know the expected JSONs to send to start/configure workflows.
Pros of a package:
- NICOS knows before the GUI starts how to render configs for different instruments and workflows and what workflows are available
- Easier for us to test that layouts make sense in pyqt
Cons of a packge:
- We cannot be as certain that we are using the same package version, for example we might still be pulling in 0.7.0, but you just redeployed with 0.8.0 which has new workflows added.
Pros of configs always in kafka:
- We can be more certain that as long as we follow the structure of the configs/worksflows in kafka, esslivedata will accept those configs
Cons of kafka:
- available config/workflows would probably be json (?) so types and such are a bit harder to make sure we follow
- potential ugly GUI side effects since we would have to dynamically create the config Qt elements. Which can be hard to forsee, also a bit harder to test offline.
What are your thoughts?
Activity
SimonHeybrock commentedon Feb 27, 2026
Great question! A long time ago we did actually publish this in Kafka, but moved away from it due to complexity etc.
Other things to consider:
essdiffractionthat define things like enums that we want to configure as part of the workflow in the dashboard. To avoid having the dashboard depend onessdiffractionwe implement a wrapper/translator inesslivedata, so the dashboard dependencies are kept minimal and generic. If we moved to a pure Kafka-based solution these unfortunate wrappers might go away?I'll think about this some more and then we can maybe have a chat some day!
SimonHeybrock commentedon Feb 27, 2026
Further Options and Distributed Systems Considerations
Options Beyond Package vs. Kafka
Option C: JSON Schema artifact (published, not a Python package)
Each release of esslivedata produces a JSON Schema artifact (e.g., a
.jsonfile per instrument) as part of CI. This is published somewhere retrievable — a git repo, an HTTP endpoint, a shared filesystem, or even a PyPI package containing only.jsonfiles (no Python code). NICOS consumes JSON Schema, not Python models.Why this is interesting: It breaks the Python coupling entirely. NICOS could be in any language. Pydantic already supports
model_json_schema(), so generation is trivial. JSON Schema is a well-understood standard with tooling for UI generation.Option D: REST/HTTP discovery endpoint
The running esslivedata backend exposes a lightweight HTTP endpoint (e.g.,
/api/v1/instruments/{name}/workflows) that returns available workflow specs as JSON Schema. This could be a tiny FastAPI sidecar or embedded in the dashboard.Why this is interesting: It's the "live" approach (like Kafka) but simpler — standard request/response, no subscription management, cacheable, easy to test with
curl.Option E: Hybrid — schema package + Kafka runtime validation
Publish a schema package (Option A or C) for offline UI generation and testing. At runtime, the backend also publishes a version identifier or schema hash on Kafka. NICOS can check "am I using a compatible schema?" before sending commands. Gives you the best of both worlds.
Patterns and Pitfalls in Distributed (Kafka-based) Systems
Schema-payload drift:
If NICOS has a schema from package version X but the backend is running version Y, parameter validation diverges silently. JSON is forgiving — extra fields are ignored, missing fields get defaults — but this can lead to subtle bugs where NICOS thinks it configured something that the backend silently dropped.
Mitigation: Include a
spec_versionor schema hash in everyWorkflowConfigmessage. The backend rejects or warns on mismatch. This is explicit contract verification, not implicit hope.Compacted topic stale data:
If specs are published to a compacted Kafka topic (so consumers always get the latest), you need careful key design. If you rename or remove a workflow, the old key stays in the compacted log forever unless you publish a tombstone. NICOS could see phantom workflows that no longer exist.
Mitigation: Use a single key per instrument (not per workflow) and publish the complete list each time. Or use a versioned "catalog" message with an explicit list, so the consumer always has the full picture.
Kafka as a service discovery mechanism:
Kafka is designed for event streaming, not service discovery. Using it to answer "what's available right now?" is awkward because: you need to consume from the beginning of the topic to get the latest state, there's no request/response (you subscribe and wait), and consumer group management adds complexity for what is essentially a config lookup. This is likely why the previous Kafka-based approach was abandoned. Kafka is great for "here's a command, process it" but poor for "tell me what you can do."
Dynamic UI generation is hard:
Dynamically creating Qt widgets from a JSON schema is fragile — layout, validation, enum rendering, nested models are all edge cases. If NICOS goes the dynamic route, invest in a well-tested schema-to-widget mapper and constrain the schema to a subset (flat params with known types, enums, ranges). The current Pydantic models are already fairly flat, which helps.
The "version mismatch window":
With any approach, there's a window between "backend deploys new version" and "NICOS learns about it." With a package, this window can be weeks (until someone updates the dependency). With Kafka, it's seconds but requires NICOS to handle dynamic schema changes at runtime. With HTTP discovery, NICOS can poll on startup and get the current truth.
Capability advertisement with heartbeat (pattern):
A clean Kafka pattern: the backend periodically publishes a "capabilities" message (every N seconds) on a well-known topic. Contains: instrument name, list of workflow IDs, schema version, and optionally full JSON schemas. Consumers don't need to read from the beginning — they just wait for the next heartbeat. If heartbeats stop, the backend is down. This avoids the compacted-topic pitfalls and gives NICOS both discovery and health monitoring in one mechanism.
Assessment
The strongest approach might be a hybrid (Option E or variant):
WorkflowConfig) so the backend can reject commands from an incompatible schema version.The pure Kafka discovery approach has real complexity costs (likely why it was abandoned before). The pure package approach has the version drift problem correctly identified in the issue. JSON Schema artifacts split the difference: they're static and testable like a package, but language-agnostic and don't require NICOS to depend on Python Pydantic models.
Note: the technique-package wrapper problem is orthogonal — that's about runtime dependencies for workflow execution, not for schema publication. The Pydantic parameter models (like
PowderWorkflowParams) already live in esslivedata, not in essdiffraction, so they're already decoupled from technique packages for schema purposes.SimonHeybrock commentedon Feb 27, 2026
@hurvan See some more options above. One key question is whether we actually need to expose available workflows in their entirety — if we implement #445 (comment) then NICOS could just use any config that the ESSlivedata dashboard published. But I am not convinced yet that that is a truly workable solution. NICOS would still need to track (and display) those configs so you can choose which one to use for running a scan?
SimonHeybrock commentedon Aug 18, 2026
Not planned currently, replaced by ADR 0006: Expose designated workflow outputs to NICOS as derived devices.