Remote Kafka and SR host support

[MSeal]

Who is affected by this issue?

• Development manager tasked with ensuring a team is successful in using streaming data
• Developer on a team tasked with building new features using streaming data
• Developer at an org who manages a streaming platform while building new features that use streaming data
• Developer tasked with building pipelines for use in business analytics
• Other: [User type] tasked with [primary goal in the realm of streaming data]

What problem are these users experiencing?

The user has Kafka and Schema Registry resources that are not local managed, nor on Confluent Cloud. They may have data stored or populated in a hosted Kafka configuration either managed by themselves or by another team. The user wants to analyze this data and interact with it in the same manner they would on a Confluent Cloud or local topic. However, without a connection, they are forced to copy the data to a Cloud or local topic (if allowed), or wait until they reach a development stage where data might be in one of those locations. Or simply not use the extension to help their development.

How will this issue solve this problem?

We will enable reaching and recalling a remote connection pairing of Kafka and/or Schema Registry under a variety of common auth patterns so that after establishing a connection they reuse the existing capabilities of the application. This enables them to connect without changing the home of their data when it's not in a currently available source for use of the extension. Additionally this can be used to connect to a subset of a Confluent Platform hosted resource ahead of better CP supporting features.

Goals

1. Enable remote Apache Kafka and Schema Registry targets for connection without an external REST proxy
2. Enable associating new AK and SR instances together
3. Handle common authentication patterns
4. Remember connection URI and settings over time
5. Pair associated connection resources both in connection definition and in the UX of presenting those together in the UI
6. Ability for user to remove remote connections that are no longer relevant
7. Related features should work the same as they do for local Kafka/SR

Non-Goals

1. Launching a remote Kafka or SR instance
2. Enabling Kafka-like endpoints (they may work, but it's not in scope for this discussion)
3. Bubbling up ACLs from remote Kafka (it may come for free but this should be a separate effort as needed)

Background

For the extension to truly support Open Source Kafka and related resources, it needs to seamlessly be able to connect and interact with those resources in remote settings. The philosophy that local Kafka should work the same as Confluent Cloud Kafka for all supported capabilities within the extension also applies to remote Kafka sources. We want to get the connection, auth, and translation details abstracted away to support getting development done without deep coupling or knowledge of a single way of hosting required services/data.

Proposal

• Implement Kafka REST Proxy interface endpoints needed for extension to speak to any native protocol Kafka/SR
• Add a new connection type to the cluster resources subpanel (needs some design first and final pass engagement)
• Add connection detail persistence to the extension via additional secret store save for this type of connection
• Add creation/edit panels for changing details
• Map out auth schemes that are needed to support this in the ide-sidecar

[MSeal]

A more traditional user story format that may help with the open UX questions:

"As a Kafka application developer I want to select Kafka and Schema Registry URI pairs along with any authentication credentials required to establish a connection."
"As a Kafka application developer with a remote URI Kafka pairing, I want to be able to switch between connection pairs I've established easily and operate with one of said pairings for all related commands and operations."
"As a Kafka application I want the VS Code extension to remember URIs and credential entry for past pairing I've not explicitly removed so I can easily reuse them in future sessions."