Skip to content

Commit

Permalink
azure-cosmos-kafka-connect skeleton (Azure#37283)
Browse files Browse the repository at this point in the history
* azure-cosmos-kafka-connect skeleton

* Build fixes

* Update .docsettings.yml

* Update FaultInjectionWithAvailabilityStrategyTests.java

* Fixing linting errors

* Linting issues

* Create package-info.java

* Update sdk/cosmos/ci.yml

Co-authored-by: Kushagra Thapar <kushuthapar@gmail.com>

* Reacting to PR review feedback

* Update README.md

* Adding shading

* Update pom.xml

* Update pom.xml

* Addressing build analyzer issues

* Update CHANGELOG.md

---------

Co-authored-by: Kushagra Thapar <kushuthapar@gmail.com>
  • Loading branch information
FabianMeiswinkel and kushagraThapar authored Oct 31, 2023
1 parent 35ef589 commit aa3a102
Show file tree
Hide file tree
Showing 14 changed files with 606 additions and 0 deletions.
2 changes: 2 additions & 0 deletions eng/.docsettings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ known_presence_issues:
- ['sdk/cognitiveservices/ms-azure-cs-visualsearch/CHANGELOG.md', '#2847']
- ['sdk/cognitiveservices/ms-azure-cs-websearch/CHANGELOG.md', '#2847']
- ['sdk/cosmos/azure-cosmos-examples/CHANGELOG.md', '#2847']
- ['sdk/cosmos/azure-cosmos-kafka-connect/CHANGELOG.md', '#2847']
- ['sdk/cosmos/CHANGELOG.md', '#2847']
- ['sdk/eventgrid/azure-messaging-eventgrid/CHANGELOG.md', '#2847']
- ['sdk/eventgrid/microsoft-azure-eventgrid/CHANGELOG.md', '#2847']
Expand Down Expand Up @@ -114,6 +115,7 @@ known_content_issues:
- ['sdk/cosmos/faq/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-benchmark/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-examples/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-kafka-connect/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-encryption/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/README.md', '#3113']
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,7 @@ the main ServiceBusClientBuilder. -->

<!-- Cosmos sdk suppressions -->
<suppress checks="[a-zA-Z0-9]*" files="[/\\]azure-cosmos-encryption[/\\]src[/\\]test[/\\]"/>
<suppress checks="[a-zA-Z0-9]*" files="[/\\]azure-cosmos-kafka-connect[/\\]src[/\\]test[/\\]"/>
<suppress checks="[a-zA-Z0-9]*"
files=".*[/\\]azure-cosmos[/\\]((?!(BridgeInternal|CosmosBridgeInternal|CommonsBridgeInternal)).*)"/>
<suppress checks="[a-zA-Z0-9]*"
Expand Down
7 changes: 7 additions & 0 deletions eng/versioning/external_dependencies.txt
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,13 @@ cosmos_org.scalatest:scalatest-maven-plugin;2.0.2
cosmos_net.alchim31.maven:scala-maven-plugin;4.8.1
cosmos_org.scalastyle:scalastyle-maven-plugin;1.0.0

## Cosmos Kafka connector under sdk\cosmos\azure-cosmos-kafka-connect\pom.xml
# Cosmos Kafka connector runtime dependencies
cosmos_org.apache.kafka:connect-api;3.6.0
# Cosmos Kafka connector tests only
# Maven Tools for Cosmos Kafka connector only
cosmos_io.confluent:kafka-connect-maven-plugin;0.12.0

# sdk\core\azure-core-serializer-avro-jackson\pom.xml
# This dependency is needed since Jackson Avro uses an older dependency on Apache Avro which is another library.
jacksonavro_org.apache.avro:avro-maven-plugin;1.8.2
Expand Down
1 change: 1 addition & 0 deletions eng/versioning/version_client.txt
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;4.23.0;4.24.0-beta.1
com.azure:azure-cosmos-encryption;2.6.0;2.7.0-beta.1
com.azure:azure-cosmos-test;1.0.0-beta.6;1.0.0-beta.7
com.azure:azure-cosmos-tests;1.0.0-beta.1;1.0.0-beta.1
com.azure.cosmos.kafka:azure-cosmos-kafka-connect;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-data-appconfiguration;1.4.10;1.5.0-beta.3
com.azure:azure-data-appconfiguration-perf;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-data-schemaregistry;1.4.0;1.5.0-beta.1
Expand Down
11 changes: 11 additions & 0 deletions sdk/cosmos/azure-cosmos-kafka-connect/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
## Release History

### 1.0.0-beta.1 (Unreleased)

#### Features Added

#### Breaking Changes

#### Bugs Fixed

#### Other Changes
88 changes: 88 additions & 0 deletions sdk/cosmos/azure-cosmos-kafka-connect/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Apache Kafka Connect source and sink connectors for Azure Cosmos DB Java SDK for Java
The Azure Cosmos DB connectors allow moving data between Azure Cosmos DB and Kafka. The Cosmos DB Sink connector writes data from a Kafka topic to a Cosmos DB container. The Cosmos DB Source connector writes changes from a Cosmos DB container to a Kafka topic.

[Source code][kafka_source_code] | [Package (Maven)][cosmos_kafka_maven] | [Product documentation][cosmos_docs]

## Getting started
### Include the package

[//]: # ({x-version-update-start;com.azure.kafka-connect:azure-cosmos-kafka-connect;current})
```xml
<dependency>
<groupId>com.azure.kafka.connect</groupId>
<artifactId>azure-cosmos-kafka-connect</artifactId>
<version>2.0.0-beta.1</version>
</dependency>
```
[//]: # ({x-version-update-end})

Refer to maven central for previous [releases][cosmos_kafka_maven]

### Prerequisites

- [Java Development Kit (JDK) with version 8 or above][jdk]
- An active Azure account. If you don't have one, you can sign up for a [free account][azure_subscription]. Alternatively, you can use the [Azure Cosmos DB Emulator](https://docs.microsoft.com/azure/cosmos-db/local-emulator) for development and testing. As emulator HTTPS certificate is self-signed, you need to import its certificate to java trusted cert store as [explained here](https://docs.microsoft.com/azure/cosmos-db/local-emulator-export-ssl-certificates)
- (Optional) SLF4J is a logging facade.
- (Optional) [SLF4J binding](https://www.slf4j.org/manual.html) is used to associate a specific logging framework with SLF4J.
- (Optional) [Maven][maven]

SLF4J is only needed if you plan to use logging, please also download an SLF4J binding which will link the SLF4J API with the logging implementation of your choice. See the [SLF4J user manual](https://www.slf4j.org/manual.html) for more information.

The SDK provides Reactor Core-based async APIs. You can read more about Reactor Core and [Flux/Mono types here](https://projectreactor.io/docs/core/release/api/)

## Key concepts

TBD

## Examples
TBD

## Troubleshooting

### General

Azure Cosmos DB is a fast and flexible distributed database that scales seamlessly with guaranteed latency and throughput.
You do not have to make major architecture changes or write complex code to scale your database with Azure Cosmos DB.
Scaling up and down is as easy as making a single API call or SDK method call.
However, because Azure Cosmos DB is accessed via network calls there are client-side optimizations you can make to achieve peak performance when using Azure Cosmos DB Java SDK v4.

- [Performance][perf_guide] guide covers these client-side optimizations.

- [Troubleshooting Guide][troubleshooting] covers common issues, workarounds, diagnostic steps, and tools when you use Azure Cosmos DB Java SDK v4 with Azure Cosmos DB SQL API accounts.

## Next steps

- Quick start of Cosmos DB core java sdk [quickstart][quickstart] - Building a java app to manage Cosmos DB SQL API data
- [Read more about Azure Cosmos DB Service][cosmos_docs]

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
[Contributor License Agreement (CLA)][cla] declaring that you have the right to, and actually do, grant us the rights
to use your contribution.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate
the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to
do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct][coc]. For more information see the [Code of Conduct FAQ][coc_faq]
or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.

<!-- LINKS -->
[kafka_source_code]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-kafka-connect/src
[cosmos_introduction]: https://docs.microsoft.com/azure/cosmos-db/
[cosmos_docs]: https://docs.microsoft.com/azure/cosmos-db/introduction
[jdk]: https://docs.microsoft.com/java/azure/jdk/
[maven]: https://maven.apache.org/
[cosmos_kafka_maven]: https://central.sonatype.com/artifact/com.azure.kafka.connect/azure-cosmos-kafka-connect
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:opencode@microsoft.com
[azure_subscription]: https://azure.microsoft.com/free/
[troubleshooting]: https://docs.microsoft.com/azure/cosmos-db/troubleshoot-java-sdk-v4-sql
[perf_guide]: https://docs.microsoft.com/azure/cosmos-db/performance-tips-java-sdk-v4-sql?tabs=api-async
[sql_api_query]: https://docs.microsoft.com/azure/cosmos-db/sql-api-sql-query
[quickstart]: https://docs.microsoft.com/azure/cosmos-db/create-sql-api-java?tabs=sync

![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-java%2Fsdk%2Fcosmos%2Fazure-cosmos-kafka-connect%2FREADME.png)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit aa3a102

Please sign in to comment.