✅ Setup and start Gemfire database (Source)
✅ Setup and start Redis Enterprise database (Target)
✅ Setup and start Redis Connect
✅ Perform Initial load and CDC with Redis Connect on Strings
✅ Perform Initial load with Redis Connect on complex data types
Docker compatible *nix OS and Docker installed.
Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect instances deployed on separate hosts.
Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts.
i.e.
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
mkdir -p redis-connect/demo && \
unzip main.zip "redis-connect-dist-main/examples/gemfire/*" -d redis-connect && \
cd redis-connect && \
cp -R redis-connect-dist-main/examples/gemfire/demo/* ./demo && \
chmod a+x demo/*.sh && \
rm -rf ./redis-connect-dist-main && rm ../main.zip && \
cd demoApache Geode on Docker
Execute setup_gemfire.sh
demo$ ./setup_gemfire.shValidate Gemfire database is running as expected:
demo$ docker ps -a | grep gemfire
beb0205a037f apachegeode/geode:1.12.9 "sh -c /geode/script…" 4 hours ago Up 4 hours 0.0.0.0:1099->1099/tcp, 0.0.0.0:7070->7070/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:10334->10334/tcp, 0.0.0.0:40404->40404/tcp gemfire-1.12.9-virag-WPG6PH3FV5
demo$ docker exec -it gemfire-1.12.9-$(hostname) sh -c "gfsh version"
1.12.9
Execute setup_re.sh
demo$ ./setup_re.shNOTE
The above script will create a 1-node Redis Enterprise cluster in a docker container, Create a target database with RediSearch module, Create a job management and metrics database with RedisTimeSeries module, Create a RediSearch index for emp Hash, Start a docker instance of grafana with Redis Data Source and Start an instance of RedisInsight.
Review options by running Redis Connect docker container
docker run \
-it --rm --privileged=true \
--name redis-connect-$(hostname) \
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \
--net host \
redislabs/redis-connectExpected output:
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf
Example environment variables and volume mapping for docker based deployments
-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL]
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL]
-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL]
-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL]
-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config
-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials
-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL]
-p 8282:8282
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
-v: Print version.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------Start Redis Connect Instance
docker run \
-it --rm --privileged=true \
--name redis-connect-$(hostname) \
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \
--net host \
redislabs/redis-connect startExpected output:
-------------------------------
Starting redis-connect v0.10.1.5 Instance using JAVA 11.0.19 on docker-desktop started by root in /opt/redislabs/redis-connect/bin
Loading redis-connect Instance configuration from /opt/redislabs/redis-connect/config/jobmanager.properties
Instance classpath /opt/redislabs/redis-connect/lib/*:/opt/redislabs/redis-connect/extlib/*
Check redis-connect-manager-<PID>.log for cluster-level information, redis-connect-heartbeat-<PID>.log for heartbeat-lease renewals, and redis-connect-<PID>.log for the job-level information
07:11:09.184 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
/####### /## /## /###### /##
| ##__ ## | ## |__/ /##__ ## | ##
| ## \ ## /###### /####### /## /####### | ## \__/ /###### /####### /####### /###### /####### /######
| #######/ /##__ ## /##__ ##| ## /##_____/ | ## /##__ ##| ##__ ##| ##__ ## /##__ ## /##_____/|_ ##_/
| ##__ ##| ########| ## | ##| ##| ###### | ## | ## \ ##| ## \ ##| ## \ ##| ########| ## | ##
| ## \ ##| ##_____/| ## | ##| ## \____ ## | ## ##| ## | ##| ## | ##| ## | ##| ##_____/| ## | ## /##
| ## | ##| #######| #######| ## /#######/ | ######/| ######/| ## | ##| ## | ##| #######| ####### | ####/
|__/ |__/ \_______/ \_______/|__/|_______/ \______/ \______/ |__/ |__/|__/ |__/ \_______/ \_______/ \___/ v0.10.1
Powered by Redis Enterprise
07:11:14.190 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
07:11:14.930 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobManager
07:11:14.934 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobReaper
07:11:14.939 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobClaimer
07:11:14.943 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobOrchestrator
07:11:14.947 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: HeartbeatManager
07:11:14.949 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: MetricsReporter
07:11:14.951 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: CredentialsRotationEventListener
07:11:14.953 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: ChangeEventQueue
07:11:14.986 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully created Job Claim Assignment Stream and Consumer Group
07:11:14.988 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobManager service
07:11:14.988 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobReaper service
07:11:14.989 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobClaimer service
07:11:14.990 [main] INFO redis-connect-manager - Instance: 30@docker-desktop Metrics are not enabled so MetricsReporter threadpool will not be instantiated
07:11:16.630 [main] INFO redis-connect-manager - Instance: 30@docker-desktop started Redis Connect REST API listening on ["http-nio-8282"]
07:11:16.630 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
07:11:16.630 [main] INFO redis-connect-manager -
07:11:16.630 [main] INFO redis-connect-manager - Started Redis Connect Instance v0.10.1
07:11:16.630 [main] INFO redis-connect-manager -
07:11:16.630 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
07:11:24.997 [JOB_MANAGER_THREADPOOL-2] INFO redis-connect-manager - Instance: 30@docker-desktop was successfully elected Redis Connect cluster leaderOpen browser to access Swagger UI - http://localhost:8282/swagger-ui/index.html
For quick start, use 'cdc_job' as jobName
Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use the sample cdc-job.json configuration: Gemfire
Or Use curl to create the cdc-job configuration
demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"
INSERT few records into Gemfire region (source)
demo$ ./load.sh
Inserting records in session region..
(1) Executing - connect --locator localhost[10334]
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=beb0205a037f, port=1099] ..
Successfully connected to: [host=beb0205a037f, port=1099]
You are connected to a cluster of version: 1.12.9
(2) Executing - put --key=(Key1) --value=(Value1) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key1}
Value Class : java.lang.String
Old Value : "{UpdatedValue1}"
(3) Executing - put --key=(Key2) --value=(Value2) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key2}
Value Class : java.lang.String
Old Value : "{UpdatedValue2}"
(4) Executing - put --key=(Key3) --value=(Value3) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key3}
Value Class : java.lang.String
Old Value : "{Value3}"
(5) Executing - put --key=(Key4) --value=(Value4) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key4}
Value Class : java.lang.String
Old Value : "{Value4}"
(6) Executing - put --key=(Key5) --value=(Value5) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key5}
Value Class : java.lang.String
Old Value : "{Value5}"
(7) Executing - put --key=(Key6) --value=(Value6) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key6}
Value Class : java.lang.String
Old Value : "{Value6}"
(8) Executing - put --key=(Key7) --value=(Value7) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key7}
Value Class : java.lang.String
Old Value : null
(9) Executing - put --key=(Key8) --value=(Value8) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key8}
Value Class : java.lang.String
Old Value : "{Value8}"
(10) Executing - put --key=(Key9) --value=(Value9) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key9}
Value Class : java.lang.String
Old Value : null
(11) Executing - put --key=(Key10) --value=(Value10) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key10}
Value Class : java.lang.String
Old Value : "{Value10}"
doneStart Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
Use 'load' as jobType
Or Use curl to start the initial load for cdc-job
demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"
Query for the above inserted record in Redis (target)
demo$ Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
Use 'stream' as jobType
Or Use curl to start the stream for cdc-job
demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/stream" -H "accept: */*"
Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus
Or Use curl to query the cdc-job status
demo$ curl -X GET "http://localhost:8282/connect/api/v1/cluster/jobs/claim/all" -H "accept: */*"
Expected output: [{"jobId":"{connect}:job:cdc-job","jobName":"cdc-job","jobStatus":"CLAIMED","jobOwner":"30@virag-cdc","jobType":"STREAM"}]
INSERT a record into Gemfire region (source)
demo$ ./insert.sh
Inserting records in session region..
(1) Executing - connect --locator localhost[10334]
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=beb0205a037f, port=1099] ..
Successfully connected to: [host=beb0205a037f, port=1099]
You are connected to a cluster of version: 1.12.9
(2) Executing - put --key=(Key11) --value=(Value11) --region=/session
Result : true
Key Class : java.lang.String
Key : {Key11}
Value Class : java.lang.String
Old Value : "{Value11}"
doneQuery for the above inserted record in Redis (target)
demo$
This demo also includes an example of using more complex Java types within Gemfire. The extlib folder contains two jar files:
gemfire-pojo-1.0.jarredis-connect-custom-stage-sample-*.jar
The first jar contains the code for a redis.gemfire.Customer class, which is a very simple POJO containing a name and age. The second jar is a build of redis-connect-custom-stage containing some custom staging code to transform arbitrary data types (in this case a Customer) into a type that redis-connect can serialize to JSON.
On http://localhost:8282/swagger-ui/index.html go to:
Create Job Configuration - /connect/api/vi/job/config/{jobName}
_For the customer POJO, use the sample customer-pojo-job.json configuration: Gemfire
Use 'customer-pojo-job' as the jobName
Or use curl to create the customer-pojo-job configuration:
$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/customer-pojo-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/customer-pojo-job.json;type=application/json"
INSERT customer records into gemfire
$ ./load_customers.sh
Inserting records in session region..
(1) Executing - connect --locator localhost[10334]
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=749a3ef94bf8, port=1099] ..
Successfully connected to: [host=749a3ef94bf8, port=1099]
You are connected to a cluster of version: 1.15.1
(2) Executing - put --key="customer1" --value=("name":"Jack","age":35) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer1
Value Class : redis.gemfire.Customer
Old Value : null
(3) Executing - put --key="customer2" --value=("name":"Alice","age":36) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer2
Value Class : redis.gemfire.Customer
Old Value : null
(4) Executing - put --key="customer3" --value=("name":"Bob","age":37) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer3
Value Class : redis.gemfire.Customer
Old Value : null
(5) Executing - put --key="customer4" --value=("name":"Carol","age":38) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer4
Value Class : redis.gemfire.Customer
Old Value : null
(6) Executing - put --key="customer5" --value=("name":"David","age":39) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer5
Value Class : redis.gemfire.Customer
Old Value : null
(7) Executing - put --key="customer6" --value=("name":"Eva","age":40) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer6
Value Class : redis.gemfire.Customer
Old Value : null
(8) Executing - put --key="customer7" --value=("name":"Frank","age":41) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer7
Value Class : redis.gemfire.Customer
Old Value : null
(9) Executing - put --key="customer8" --value=("name":"Grace","age":42) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer8
Value Class : redis.gemfire.Customer
Old Value : null
(10) Executing - put --key="customer9" --value=("name":"Henry","age":43) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer9
Value Class : redis.gemfire.Customer
Old Value : null
(11) Executing - put --key="customer10" --value=("name":"Ivy","age":44) --region=customer --value-class=redis.gemfire.Customer
Result : true
Key Class : java.lang.String
Key : customer10
Value Class : redis.gemfire.Customer
Old Value : null
doneStart Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
Use 'load' as jobType and 'customer-pojo-job' as the jobName
Or Use curl to start the initial load for customer-pojo-job
$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/customer-pojo-job/load" -H "accept: */*"
Your data is now available in Redis at the keys {gemfireRegionName}:{gemfireKeyName} and can be queried in Redis using the redis-cli:
$ redis-cli -p 14000 JSON.GET customer:customer1 $
"[{\"name\":\"Jack\",\"age\":35}]"