-
Notifications
You must be signed in to change notification settings - Fork 186
Description
Is your feature request related to a problem? Please describe.
During the release process, several Kafka issues inevitably come up. This is mostly due to the difficulty testing Kafka in any automated way. In CI, options for creating additional services is limited and mocking Kafka would be very difficult. The result is most of our unit tests only read/write to files leaving pipelines that use Kafka untested.
Describe the solution you'd like
Ideally, we would integrate Kafka into end-to-end tests that can be run by CI (Tracking issue #258). Until that issue is completed, we will need to manually test kafka workflows. A rough outline of how this would be completed:
- Every 2 weeks, a rotating person will download the latest development branch
- Create a local Kafka cluster
- Run all supported workflows using Kafka
- Create any issues for bugs that are discovered.
- Update any out of date documentation
Describe alternatives you've considered
See issue #258
Additional context
Documentation for the above steps will need to be created as part of this issue.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status