feat!: inlined schema definitions inside the KafkaSchema custom resource

[Tomek-Adamczewski]

v1beta1 approach was based on the convention that the actual schema (payload/structure) was stored in configmap and only referenced by KafkaSchema resource. There are a few problems related to that approach:

1. inconsistent behaviour - when you create a schema resource, configmap needs to be already applied in your namespace. but finalizer deletes that configmap during cleanup
2. race condition between K8S resources - during reconciliation, it's assumed that configmap was updated prior to updating KafkaSchema. If it wasn't, changes will not be applied. On the other hand, configmap has to be deleted after the custom resource, otherwise finalizer will fail.
3. difficulties in managing "independent" K8S resources in a single chart - it's not easy/recommended to use owner references from within a single chart, and playing with finalizers to "block deletion" of configmap is tricky

Instead, I would like to propose a simpler approach, where the entire KafkaSchema resource is self-contained. You can define your schema payload directly in the resource or use helm templating to populate it from an external file

BREAKING CHANGE: CR no longer supports reference to schema configmap through spec.data.configRef

[slomkarafa]

halo, greate work!

[slomkarafa]

LGTM!

[dynamike2010]

I've been waiting for this - thank you!

[Brightside56]

Inline definitions are ok, but plaintext credentials for schema registry in CR don't make me happy 😄 There could be registryCredentialsSecretRef at least

[Tomek-Adamczewski]

@Brightside56 agreed. I'll fix that in another PR. Please don't use it for the time being ;)

[slomkarafa]

@Brightside56 that is just a comment or you need to use credentials in your use case? because we wanted to drop authentication feature for now, as there is a plan to do it in more sophisticated way to allow different auth methods, not only Basic

[Brightside56]

@slomkarafa Yes, I need to use credentials and I need them to be processed securely

[Brightside56]

@Brightside56 agreed. I'll fix that in another PR. Please don't use it for the time being ;)

For now it's more than ok. I'm concerned more with issues like absence of schema validation and possibility to check resources health using CR status

[slomkarafa]

@Brightside56 and possibility to check resources health using CR status that's already planned, but we (or mainly @Tomek-Adamczewski :p) are doing that in the meantime, so we strongly encourage to contribution :p

[Tomek-Adamczewski]

@Brightside56 on schema validation - schema registry should do that for you. do you think we should duplicate that responsibility in operator?

Anyway, I've enabled issues in this repo for you to start throwing darts at me. When I think what this operator should actually do, I'm considering creating a completely new api group and start from scratch, as I won't be able to maintain any compatibility.