Skip to content

[FEA] Create Kafka Testing Procedure #265

@mdemoret-nv

Description

@mdemoret-nv

Is your feature request related to a problem? Please describe.
During the release process, several Kafka issues inevitably come up. This is mostly due to the difficulty testing Kafka in any automated way. In CI, options for creating additional services is limited and mocking Kafka would be very difficult. The result is most of our unit tests only read/write to files leaving pipelines that use Kafka untested.

Describe the solution you'd like
Ideally, we would integrate Kafka into end-to-end tests that can be run by CI (Tracking issue #258). Until that issue is completed, we will need to manually test kafka workflows. A rough outline of how this would be completed:

  1. Every 2 weeks, a rotating person will download the latest development branch
  2. Create a local Kafka cluster
  3. Run all supported workflows using Kafka
  4. Create any issues for bugs that are discovered.
  5. Update any out of date documentation

Describe alternatives you've considered
See issue #258

Additional context
Documentation for the above steps will need to be created as part of this issue.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions