Automated Data Generation

[MSeal]

Goals

1. Enable data generation to a topic from within the extension interface
2. Schema aware, or possible required, to enable relations between fields and realistic value attributions
3. Controls around the rate or duration of event generation
4. No code required for default usecase
5. [Idea] Enable test frameworks to utilize library for data gen?
6. [Idea] Enable rehydration of the same set of generated messages to a topic?

Non-Goals

1. Production data manipulation or population
2. Complex expressions of data artifacts (keep the builder patterns simple / within reason)

Background

Launching new topics for development often results in empty message queues that require creating a producer program or performing some sort of data sanitization and sync from real sources. Populating these automatically, especially from schema structures, would be tremendously valuable in project development initialization and testing.

Proposal

Ideally there would be a call to action on each topic row to generate data. The call to action could ask some basic questions around rate of messages, duration of bursts, schema to use (default to associated schemas when possible), etc. The sidecar should be the actor that runs the generation and population effort to keep the typescript UI management code simpler and to enable this for scenarios that require different API patterns or authentication. The sidecar would need to expose a datagen API that is populated by the aforementioned fields. This would likely be a fire-and-forget route given the nature of long generation requests. That would mean there would also need to be a generation halt route (perhaps just a general kill-all-generators for simplicity).

TBD on if API key management is required to achieve this effort.
TBD on exact tool used for the individual message generation (there's some tools already built by Confluent for customers and some in open source that could be utilized)

This is in scope for the Confluent team to explore in the near term.