Skip to content

Commit df64a93

Browse files
JaidenAshmoreJaidenAshmore
authored
140 spring cloud registry extension (#146)
Adds the ability to deserialize messages that have been sent using a schema like Avro which is registered in the Spring Cloud Schema Registry.
1 parent 68fb92a commit df64a93

File tree

65 files changed

+2586
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+2586
-0
lines changed

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,11 @@ If the Spring Boot application needs to connect to SQS queues across multiple AW
239239
which will be able to obtain a specific `SqsAsyncClient` based on an identifier. For more information on how to do this, take a look at the documentation
240240
at [How To Connect to Multiple AWS Accounts](doc/how-to-guides/spring/spring-how-to-connect-to-multiple-aws-accounts.md)
241241

242+
### Versioning Message Payloads using Apache Avro Schemas
243+
As the application grows, it may be beneficial to allow for versioning of the schema so that the consumer can still serialize messages from producers sending
244+
different versions of the schema. To allow for this the [spring-cloud-schema-registry-extension](extensions/spring-cloud-schema-registry-extension) was written
245+
to support this functionality. See the [README.md](extensions/spring-cloud-schema-registry-extension/README.md) for this extension for more details.
246+
242247
### Comparing Libraries
243248
If you want to see the difference between this library and others like the
244249
[Spring Cloud AWS Messaging](https://github.com/spring-cloud/spring-cloud-aws/tree/master/spring-cloud-aws-messaging) and

‎configuration/spotbugs/bugsExcludeFilter.xml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,7 @@
44
<Match>
55
<Bug pattern="SE_NO_SERIALVERSIONID" />
66
</Match>
7+
<Match>
8+
<Source name="~.*generated-sources.*" />
9+
</Match>
710
</FindBugsFilter>

‎doc/documentation.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ more in depth understanding take a look at the JavaDoc for the [java-dynamic-sqs
3232
processing of messages for specific queue listeners
3333
1. [How to connect to multiple AWS Accounts](how-to-guides/spring/spring-how-to-connect-to-multiple-aws-accounts.md): guide for listening to queues
3434
across multiple AWS Accounts
35+
1. [How to version message payload schemas](how-to-guides/spring/spring-how-to-version-payload-schemas-using-spring-cloud-schema-registry.md): guide
36+
for versioning payloads using Avro and the Spring Cloud Schema Registry.
3537
1. Local Development:
3638
1. [Setting up IntelliJ](local-development/setting-up-intellij.md): steps for setting IntelliJ up for development,
3739
e.g. configuring checkstyle, Lombok, etc

‎doc/how-to-guides/spring/spring-how-to-version-payload-schemas-using-spring-cloud-schema-registry.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Spring - How to version message payload Schemas using Spring Cloud Schema Registry
2+
As your application grows over time the format of the data that needs to be sent in the SQS messages may change as well. To allow for
3+
these changes, the [Spring Cloud Schema Registry](https://cloud.spring.io/spring-cloud-static/spring-cloud-schema-registry/1.0.0.RC1/reference/html/spring-cloud-schema-registry.html)
4+
can be used to track the version of your schemas, allowing the SQS consumer to be able to interpret multiple versions of your payload.
5+
6+
## Full reference
7+
For a full working solution of this feature, take a look at the [Spring Cloud Schema Registry Example](../../../examples/spring-cloud-schema-registry-example).
8+
9+
## Steps to consume messages serialized using Apache Avro
10+
1. Include the `Spring Cloud Schema Registry Extension` dependency
11+
```xml
12+
<dependency>
13+
<groupId>com.jashmore</groupId>
14+
<artifactId>avro-spring-cloud-schema-registry-extension</artifactId>
15+
<version>${project.version}</version>
16+
</dependency>
17+
```
18+
1. Define your schemas and map this in your spring `application.yml`
19+
```yml
20+
spring:
21+
cloud:
22+
schema-registry-client:
23+
endpoint: http://localhost:8990
24+
schema:
25+
avro:
26+
schema-imports:
27+
- classpath:avro/author.avsc
28+
schema-locations:
29+
- classpath:avro/book.avsc
30+
```
31+
In this example above we have a book schema which is dependent on the author schema. We have also hardcoded the Schema Registry
32+
to be at [http://localhost:8990](http://localhost:8990).
33+
1. Create your schemas and place them in your `resources` directory. For example this is an example schema for the Book.
34+
```json
35+
{
36+
"namespace" : "com.jashmore.sqs.extensions.registry.model",
37+
"type" : "record",
38+
"name" : "Book",
39+
"fields" : [
40+
{ "name":"id","type":"string" },
41+
{ "name":"name","type":"string" },
42+
{ "name":"author","type":"Author" }
43+
]
44+
}
45+
```
46+
1. Enable the extension by annotating the Spring Application
47+
```java
48+
@EnableSchemaRegistrySqsExtension
49+
@SpringBootApplication
50+
class Application {
51+
// normal code
52+
}
53+
```
54+
1. Define your queue listener using the `@SpringCloudSchemaRegistryPayload` to represent the payload that needs to be deserialized from
55+
the message payload.
56+
```java
57+
@QueueListener(value = "queueName")
58+
public void listen(@SpringCloudSchemaRegistryPayload Book payload) {
59+
log.info("Payload: {}", payload);
60+
}
61+
```
62+
63+
## Steps to produce messages using Avro
64+
You can wrap your `SqsAsyncClient` with the
65+
[AvroSchemaRegistrySqsAsyncClient](../../../util/proxy-method-interceptor/src/main/java/com/jashmore/sqs/registry/AvroSchemaRegistrySqsAsyncClient.java)
66+
to be able to more easily send a message that will be serialized using the Avro Schema. This Avro SQS Client was built for testing purposes and therefore it is
67+
recommended to developer your own logic for sending these messages.
68+
69+
For a full example of building this client, take a look at the
70+
[Producer Example](../../../examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-producer).

‎examples/pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
<modules>
2222
<module>core-examples</module>
2323
<module>spring-aws-example</module>
24+
<module>spring-cloud-schema-registry-example</module>
2425
<module>spring-integration-test-example</module>
2526
<module>spring-multiple-aws-account-example</module>
2627
<module>spring-starter-examples</module>

‎examples/spring-cloud-schema-registry-example/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Spring Cloud Schema Registry Extension Example
2+
This example shows how you can consume messages which have been defined using an [Avro](https://avro.apache.org/docs/1.9.2/gettingstartedjava.html)
3+
Schema and the [Spring Cloud Schema Registry](https://cloud.spring.io/spring-cloud-static/spring-cloud-schema-registry/1.0.0.RC1/reference/html/spring-cloud-schema-registry.html).
4+
5+
To find the corresponding code look in the [Spring Cloud Schema Registry Extension](../../extensions/spring-cloud-schema-registry-extension) module.
6+
7+
## Steps
8+
Start each of these applications in new terminals/your IDE:
9+
1. A Spring Cloud Schema Registry server
10+
```bash
11+
wget -O /tmp/schema-registry-server.jar https://repo.spring.io/libs-release-ossrh-cache/org/springframework/cloud/spring-cloud-schema-registry-server/1.0.3.RELEASE/spring-cloud-schema-registry-server-1.0.3.RELEASE.jar
12+
cd /tmp
13+
java -jar schema-registry-server.jar
14+
```
15+
1. A local SQS server using ElasticMQ
16+
```bash
17+
docker run -p 9324:9324 softwaremill/elasticmq
18+
```
19+
1. The SQS consumer service
20+
```bash
21+
cd java-dynamic-sqs-listener/examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-consumer
22+
mvn spring-boot:run
23+
```
24+
1. The first SQS producer service
25+
```bash
26+
cd java-dynamic-sqs-listener/examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-producer
27+
mvn spring-boot:run
28+
```
29+
1. The second SQS producer service
30+
```bash
31+
cd java-dynamic-sqs-listener/examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-producer-2
32+
mvn spring-boot:run
33+
```
34+
35+
You should now see the consumer receiving messages from both producers and even though the producers are sending
36+
the payload in different schema versions the consumer is still able to process the message.

‎examples/spring-cloud-schema-registry-example/pom.xml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<parent>
6+
<artifactId>examples</artifactId>
7+
<groupId>com.jashmore</groupId>
8+
<version>3.0.1-SNAPSHOT</version>
9+
</parent>
10+
<modelVersion>4.0.0</modelVersion>
11+
12+
<artifactId>spring-cloud-schema-registry-example</artifactId>
13+
<packaging>pom</packaging>
14+
15+
<name>Java Dynamic SQS Listener - Spring Starter - Spring Cloud Schema Registry Example</name>
16+
<description>Contains examples for serializing messages using Avro and storing these Schema Definitions in the Spring Cloud Schema Registry</description>
17+
18+
<modules>
19+
<module>spring-cloud-schema-registry-consumer</module>
20+
<module>spring-cloud-schema-registry-producer</module>
21+
<module>spring-cloud-schema-registry-producer-two</module>
22+
</modules>
23+
24+
</project>

‎examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-consumer/pom.xml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<parent>
6+
<artifactId>spring-cloud-schema-registry-example</artifactId>
7+
<groupId>com.jashmore</groupId>
8+
<version>3.0.1-SNAPSHOT</version>
9+
</parent>
10+
<modelVersion>4.0.0</modelVersion>
11+
12+
<artifactId>spring-cloud-schema-registry-consumer</artifactId>
13+
14+
<name>Java Dynamic SQS Listener - Spring Starter - Spring Cloud Schema Registry Example - Consumer</name>
15+
<description>Contains an example of a consumer deserializing a message payload that is in a schema registered in the Spring Cloud Schema Registry.</description>
16+
17+
<properties>
18+
<spotbugs.config.location>../../../configuration/spotbugs/bugsExcludeFilter.xml</spotbugs.config.location>
19+
</properties>
20+
21+
<dependencies>
22+
<dependency>
23+
<groupId>org.springframework.boot</groupId>
24+
<artifactId>spring-boot-starter</artifactId>
25+
</dependency>
26+
27+
<dependency>
28+
<groupId>org.springframework.boot</groupId>
29+
<artifactId>spring-boot-autoconfigure-processor</artifactId>
30+
</dependency>
31+
32+
<dependency>
33+
<groupId>com.jashmore</groupId>
34+
<artifactId>java-dynamic-sqs-listener-spring-starter</artifactId>
35+
<version>${project.version}</version>
36+
</dependency>
37+
38+
<dependency>
39+
<groupId>org.projectlombok</groupId>
40+
<artifactId>lombok</artifactId>
41+
<scope>provided</scope>
42+
</dependency>
43+
44+
<dependency>
45+
<groupId>com.jashmore</groupId>
46+
<artifactId>avro-spring-cloud-schema-registry-extension</artifactId>
47+
<version>${project.version}</version>
48+
</dependency>
49+
50+
<dependency>
51+
<groupId>com.jashmore</groupId>
52+
<artifactId>local-amazon-sqs</artifactId>
53+
<version>${project.version}</version>
54+
</dependency>
55+
</dependencies>
56+
57+
<build>
58+
<plugins>
59+
<plugin>
60+
<groupId>org.codehaus.mojo</groupId>
61+
<artifactId>build-helper-maven-plugin</artifactId>
62+
<executions>
63+
<execution>
64+
<id>add-source</id>
65+
<phase>generate-sources</phase>
66+
<goals>
67+
<goal>add-source</goal>
68+
</goals>
69+
<configuration>
70+
<sources>
71+
<source>${project.build.directory}/generated-sources/avro</source>
72+
</sources>
73+
</configuration>
74+
</execution>
75+
</executions>
76+
</plugin>
77+
<plugin>
78+
<groupId>org.springframework.boot</groupId>
79+
<artifactId>spring-boot-maven-plugin</artifactId>
80+
<configuration>
81+
<mainClass>com.jashmore.sqs.examples.schemaregistry.ConsumerApplication</mainClass>
82+
</configuration>
83+
<executions>
84+
<execution>
85+
<goals>
86+
<goal>repackage</goal>
87+
</goals>
88+
</execution>
89+
</executions>
90+
</plugin>
91+
<plugin>
92+
<groupId>org.apache.avro</groupId>
93+
<artifactId>avro-maven-plugin</artifactId>
94+
<executions>
95+
<execution>
96+
<phase>generate-sources</phase>
97+
<goals>
98+
<goal>schema</goal>
99+
<goal>protocol</goal>
100+
<goal>idl-protocol</goal>
101+
</goals>
102+
<configuration>
103+
<sourceDirectory>src/main/resources/avro</sourceDirectory>
104+
</configuration>
105+
</execution>
106+
</executions>
107+
</plugin>
108+
</plugins>
109+
</build>
110+
</project>

‎examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-consumer/src/main/java/com/jashmore/sqs/examples/schemaregistry/ConsumerApplication.java‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
package com.jashmore.sqs.examples.schemaregistry;
2+
3+
import com.example.Sensor;
4+
import com.jashmore.sqs.extensions.registry.SpringCloudSchemaRegistryPayload;
5+
import com.jashmore.sqs.extensions.registry.avro.EnableSchemaRegistrySqsExtension;
6+
import com.jashmore.sqs.spring.container.basic.QueueListener;
7+
import com.jashmore.sqs.util.LocalSqsAsyncClient;
8+
import com.jashmore.sqs.util.SqsQueuesConfig;
9+
import lombok.extern.slf4j.Slf4j;
10+
import org.springframework.boot.SpringApplication;
11+
import org.springframework.boot.autoconfigure.SpringBootApplication;
12+
import org.springframework.context.annotation.Bean;
13+
import software.amazon.awssdk.services.sqs.SqsAsyncClient;
14+
15+
@Slf4j
16+
@SpringBootApplication
17+
@EnableSchemaRegistrySqsExtension
18+
@SuppressWarnings("checkstyle:javadocmethod")
19+
public class ConsumerApplication {
20+
21+
public static void main(String[] args) {
22+
SpringApplication.run(ConsumerApplication.class);
23+
}
24+
25+
@Bean
26+
public SqsAsyncClient sqsAsyncClient() {
27+
return new LocalSqsAsyncClient(SqsQueuesConfig.builder()
28+
.sqsServerUrl("http://localhost:9324")
29+
.queue(SqsQueuesConfig.QueueConfig.builder()
30+
.queueName("test")
31+
.deadLetterQueueName("test-dlq")
32+
.maxReceiveCount(3)
33+
.build())
34+
.build());
35+
}
36+
37+
@QueueListener(value = "test", identifier = "message-listener")
38+
public void listen(@SpringCloudSchemaRegistryPayload Sensor payload) {
39+
log.info("Payload: {}", payload);
40+
}
41+
}

‎examples/spring-cloud-schema-registry-example/spring-cloud-schema-registry-consumer/src/main/resources/application.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
spring:
2+
cloud:
3+
schema-registry-client:
4+
endpoint: http://localhost:8990
5+
schema:
6+
avro:
7+
schema-locations:
8+
- classpath:avro/sensor.avsc

0 commit comments

Comments
 (0)