Skip to content

Kafka Streams application to populate the epics2kafka epics-channels using JAWS registered alarms

License

Notifications You must be signed in to change notification settings

JeffersonLab/jaws-registrations2epics

Repository files navigation

jaws-registrations2epics CI Docker Image Version (latest semver)

A Kafka Streams application to continuously populate the epics2kafka epics-channels topic from the JAWS alarm-instances topic for the subset of messages containing source type EPICSSource.



Overview

This app keeps epics2kafka automatically configured based on the JAWS configuration. This Kafka Streams app consumes the alarm-instances topic and looks for registration messages related to EPICS and for each of those produces a command message to epics2kafka via the epics-channels topic.

When a JAWS registration is removed it is removed via a tombstone message, which is a null value for a given key. This presents a challenge as a registration removal does not have a value payload to indicate whether it is an EPICS registration or what is the PV/channel name. The epics2kafka epics-channels topic key contains a PV/channel whereas the JAWS alarm-instances key is an alarm name. To overcome this challenge this app is not stateless, it uses a Kafka Streams store to track all JAWS registration records that have been used to command epics2kafka. This way the registration record key, the alarm name, can be used in the tombstone case to lookup if the tombstone means action is needed.

Quick Start with Compose

  1. Grab project
git clone https://github.com/JeffersonLab/jaws-registrations2epics
cd jaws-registrations2epics
  1. Launch Compose
docker compose up
  1. Monitor the epics-channels command topic for updated channels to monitor
docker exec kafka kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic epics-channels --from-beginning --property "print.key=true" --property "key.separator==" 
  1. Register an alarm
docker exec -it cli set_alarm alarm1 --pv channel1 

See: Docker Compose Strategy

Install

This application requires a Java 11+ JVM and standard library to run.

Download from Releases or build yourself.

Start scripts are created and dependencies collected by the Gradle distribution target:

gradlew assembleDist

Launch with:

UNIX:

bin/jaws-regisrations2epics

Windows:

bin/jaws-registrations2epics.bat

Configure

Environment Variables

Name Description
BOOTSTRAP_SERVERS Comma-separated list of host and port pairs pointing to a Kafka server to bootstrap the client connection to a Kafka Cluser; example: kafka:9092
SCHEMA_REGISTRY URL to Confluent Schema Registry; example: http://registry:8081

Build

This project is built with Java 17 (compiled to Java 11 bytecode), and uses the Gradle 7 build tool to automatically download dependencies and build the project from source:

git clone https://github.com/JeffersonLab/registrations2epics
cd registrations2epics
gradlew build

Note: If you do not already have Gradle installed, it will be installed automatically by the wrapper script included in the source

Note for JLab On-Site Users: Jefferson Lab has an intercepting proxy

See: Docker Development Quick Reference

Develop

In order to iterate rapidly when making changes it's often useful to run the app directly on the local workstation, perhaps leveraging an IDE. In this scenario run the service dependencies with:

docker compose -f deps.yaml up

Then run the app with:

gradlew run

Note: The STATE_DIR config is set to the gradle build dir such that running a clean task will clear the local state. You may need to reset the Kafka server state after running a clean task by restarting Kafka from scratch else using the Reset Tool.

Note: Javadocs can be generated with the command:

gradlew javadoc

Release

  1. Bump the version number in the VERSION file and commit and push to GitHub (using Semantic Versioning).
  2. The CD GitHub Action should run automatically invoking:
    • The Create release GitHub Action to tag the source and create release notes summarizing any pull requests. Edit the release notes to add any missing details. A zip file artifact is attached to the release.
    • The Publish docker image GitHub Action to create a new demo Docker image.

See Also