|
1 | | -# Redis Connect for DB2 in K8s |
2 | | - |
3 | | -This repository describes the steps involved to deploy Redis Connect for DB2 in K8s. |
4 | | - |
5 | | -Overall flow: |
6 | | -1. Clone the Redis Connect for db2 repository. |
7 | | -2. Configure Redis Connect as in <a href="../demo/config/samples" target="_blank">this set of docs</a>. |
8 | | -3. Deploy the Redis Connect configuration to Kubernetes. |
9 | | -4. Configure the Redis Connect deployment manifests. |
10 | | -5. Stage the Redis Connect job. |
11 | | -6. Start the Redis Connect job. |
12 | | - |
13 | | -**Note:** This doc uses `kubectl` and `oc` interchangeably. |
14 | | - |
15 | | -## 1. Clone the Redis Connect for db2 Repository |
16 | | - |
17 | | -Follow the [demo](../demo) steps then goto k8s-docs directory |
18 | | -``` |
19 | | -redis-connect$ cd k8s-docs |
20 | | -``` |
21 | | - |
22 | | -## 2. Configure Redis Connect |
23 | | - |
24 | | -Configure the files to describe your Redis Connect Job. One sample configuration is <a href="../demo/config/samples" target="_blank">here</a>. |
25 | | - |
26 | | -Redis Connect is a Java application which is a client of both the source RDBMS and the target Redis. As such, you will need: |
27 | | -* Source database details (endpoint, port, credentials) |
28 | | - * <a href="../" target="_blank">WAL and replication configuration</a> completed on the source database system |
29 | | -* Source schema details |
30 | | -* Target Redis details and instances (one for the data, one for the Job configuration) |
31 | | - |
32 | | -Details for configuring Redis Connect for db2 are <a href="../demo/" target="_blank">here</a>. |
33 | | - |
34 | | -## 3. Deploy the Redis Connect Configuration to Kubernetes |
35 | | - |
36 | | -This deployment requires the use of K8s ConfigMaps. The necessary config maps will be uploaded from your local directories using the commands below. |
37 | | - |
38 | | -``` |
39 | | -$ cd ../demo/config/samples/db2 |
40 | | -demo/config/samples/db2$ ls |
41 | | -FormatterConfig.yml JobManager.yml env.yml templates/ |
42 | | -JobConfig.yml Setup.yml mappers/ |
43 | | -``` |
44 | | - |
45 | | -Here is an example of creating the ConfigMap. This command should be *run from the directory containing your config files*. |
46 | | -``` |
47 | | -kubectl create configmap redis-connect-config \ |
48 | | - --from-file=JobConfig.yml=JobConfig.yml \ |
49 | | - --from-file=JobManager.yml=JobManager.yml \ |
50 | | - --from-file=env.yml=env.yml \ |
51 | | - --from-file=Setup.yml=Setup.yml \ |
52 | | - --from-file=mapper1.yml=mappers/mapper1.yml \ |
53 | | - --from-file=TaskCreator.yml=TaskCreator.yml |
54 | | -``` |
55 | | -The outcome is: |
56 | | -``` |
57 | | -$ oc get configmap/redis-connect-config |
58 | | -NAME DATA AGE |
59 | | -redis-connect-config 5 5s |
60 | | -``` |
61 | | - |
62 | | -If you need to add a custom stage jar file then you can append that to the ConfigMap creation as follows: |
63 | | -``` |
64 | | -kubectl create configmap redis-connect-config \ |
65 | | - --from-file=JobConfig.yml=JobConfig.yml \ |
66 | | - --from-file=JobManager.yml=JobManager.yml \ |
67 | | - --from-file=env.yml=env.yml \ |
68 | | - --from-file=Setup.yml=Setup.yml \ |
69 | | - --from-file=mapper1.yml=mappers/mapper1.yml \ |
70 | | - --from-file=TaskCreator.yml=TaskCreator.yml \ |
71 | | - --from-file=redis-connect-custom-stage-demo-1.0-SNAPSHOT.jar=redis-connect-custom-stage-demo-1.0-SNAPSHOT.jar |
72 | | -``` |
73 | | -The outcome is: |
74 | | -``` |
75 | | -$ oc get configmap/redis-connect-config |
76 | | -NAME DATA AGE |
77 | | -redis-connect-config 6 12s |
78 | | -``` |
79 | | - |
80 | | -If the ConfigMap did not get created, it's likely that one of more of the source configuration files was not found (eg. JobConfig.yml) so please verify the path to your files. |
81 | | - |
82 | | -The details of the the command above are: |
83 | | -`kubectl create configmap <configmap_name> --from-file=<key_name>=<path-to/file_name>` |
84 | | - |
85 | | -**Note:** ConfigMaps are immutable so if you are making changes to an existing configuration, you need to delete the existing configuration first. |
86 | | - |
87 | | -### How Does the ConfigMap get used? |
88 | | - |
89 | | -The values of keys in the ConfigMap will be mounted directly to the pod's filesystem. |
90 | | - |
91 | | -The following volume mount is defined in the manifests. The a will mount the resource `config-volume` to that `mountPath`. |
92 | | -``` |
93 | | - volumeMounts: |
94 | | - - name: config-volume |
95 | | - mountPath: /opt/redislabs/redis-connect/config/fromconfigmap |
96 | | -``` |
97 | | -The volume is defined through the following `volume` directive. It will mount the file/pth `JobConfig.yml` using the contents of the key named `JobConfig.yml` from the ConfigMap `redis-connect-config` in the `mountPath` define above. |
98 | | -``` |
99 | | - volumes: |
100 | | - - name: config-volume |
101 | | - configMap: |
102 | | - name: redis-connect-config |
103 | | - items: |
104 | | - - key: JobConfig.yml |
105 | | - path: JobConfig.yml |
106 | | -``` |
107 | | -The effect of this mapping in the pod's filesystem is the following: |
108 | | -``` |
109 | | -root@redis-connect-7b7ccf87b9-sqshl> pwd |
110 | | -/opt/redislabs/redis-connect/config/fromconfigmap |
111 | | -root@redis-connect-7b7ccf87b9-sqshl> ls -al |
112 | | -total 0 |
113 | | -drwxrwxrwx 3 root root 149 Aug 12 15:52 . |
114 | | -drwxr-xr-x 1 root root 27 Aug 12 15:52 .. |
115 | | -drwxr-xr-x 3 root root 96 Aug 12 15:52 ..2021_08_12_15_52_06.258096011 |
116 | | -lrwxrwxrwx 1 root root 31 Aug 12 15:52 ..data -> ..2021_08_12_15_52_06.258096011 |
117 | | -lrwxrwxrwx 1 root root 20 Aug 12 15:52 JobConfig.yml -> ..data/JobConfig.yml |
118 | | -lrwxrwxrwx 1 root root 21 Aug 12 15:52 JobManager.yml -> ..data/JobManager.yml |
119 | | -lrwxrwxrwx 1 root root 16 Aug 12 15:52 Setup.yml -> ..data/Setup.yml |
120 | | -lrwxrwxrwx 1 root root 14 Aug 12 15:52 env.yml -> ..data/env.yml |
121 | | -lrwxrwxrwx 1 root root 14 Aug 12 15:52 mappers -> ..data/mappers |
122 | | -``` |
123 | | -The final link is the environment variable that instructs Redis Connect to use these mapped files: |
124 | | -``` |
125 | | - env: |
126 | | - - name: REDISCONNECT_CONFIG |
127 | | - value: "/opt/redislabs/redis-connect/config/fromconfigmap" |
128 | | -``` |
129 | | - |
130 | | -## 4. Configure the Redis Connect Deployment Manifests |
131 | | - |
132 | | -Update both the `redis-connect-stage.yaml` and `redis-connect-start.yaml` to map the appropriate environment variables in the `env:` section. Notable, the `REDISCONNECT_SOURCE_USERNAME`, `REDISCONNECT_SOURCE_PASSWORD`, `REDISCONNECT_TARGET_USERNAME` and `REDISCONNECT_TARGET_PASSWORD`. |
133 | | - |
134 | | -Examples are provided to populate environment variables from the manifest/yaml, from configmap, and from k8s secrets for sensitive info such as credentials: |
135 | | - |
136 | | -``` |
137 | | - - name: REDISCONNECT_SOURCE_PASSWORD |
138 | | - value: admin123 |
139 | | - # valueFrom: |
140 | | - # configMapKeyRef: |
141 | | - # key: REDISCONNECT_SOURCE_PASSWORD |
142 | | - # name: redis-connect-config |
143 | | - # valueFrom: |
144 | | - # secretKeyRef: |
145 | | - # key: password |
146 | | - # name: redis-connect-secret |
147 | | -``` |
148 | | - |
149 | | -## 5. Stage the Redis Connect Job |
150 | | - |
151 | | -Apply the stage manifest as follows: `oc apply -f redis-connect-stage.yaml`. The outcome will be a k8s batch/Job which will run once and exit. |
152 | | -``` |
153 | | -$ oc get po -w |
154 | | -NAME READY STATUS RESTARTS AGE |
155 | | -redis-connect-stage-lkvp2 0/1 Completed 0 44s |
156 | | -``` |
157 | | - |
158 | | -The effect of this stage operation is the configuration keys are loaded in to the target Redis instance defined in `env.yml`:`jobConfigConnection`. The Job should have an `UNASSIGNED` owner. |
159 | | - |
160 | | -## 6. Start the Redis Connect Job |
161 | | - |
162 | | -Apply the stage manifest as follows: `oc apply -f redis-connect-start.yaml`. The outcome will be a k8s apps/Deployment which will run continually. |
163 | | -``` |
164 | | -$ oc get po -w |
165 | | -NAME READY STATUS RESTARTS AGE |
166 | | -redis-connect-cbc7dcd9d-k9mxj 1/1 Running 0 4s |
167 | | -redis-connect-stage-lkvp2 0/1 Completed 0 3m10s |
168 | | -``` |
169 | | - |
170 | | -The effect of the above is that the Redis Connect job has started. The Job Owner should be specified in the in `env.yml`:`jobConfigConnection` Redis DB as `JC-xx@redis-connect-cbc7dcd9d-k9mxj` indicating that the pod created is the Redis Connect job owner. You should see your changes propagate to the `targetConnection` Redis database as defined in `env.yml`. |
171 | | - |
172 | | ---- |
173 | | -### Troubleshooting Options |
174 | | - |
175 | | -1. Tail the pod logs. `oc logs -f pod/redis-connect-cbc7dcd9d-k9mxj` |
176 | | -2. Start the pods in interactive mode. |
177 | | - * Launch the pods in a do/while loop instead of the redisconnect.sh start command: |
178 | | - ``` |
179 | | - #### uncomment the following two lines while you are setting up your |
180 | | - command: [ "/bin/bash", "-c", "--" ] |
181 | | - args: [ "while true; do sleep 30; done;" ] |
182 | | - #### |
183 | | - # comment out the default starting point |
184 | | - # command: ["/opt/redislabs/redis-connect/bin/redisconnect.sh", "start"] |
185 | | - ``` |
186 | | - * Now you can leverage the `bin/redisconnect.sh` enterpoint interactively to: |
187 | | - * Test source and target connections |
188 | | - * Run Redis Connect interactively to test a configuration |
189 | | -3. Enable more verbose logging. |
190 | | - * Adjust and add `logback.xml` to your ConfigMap |
191 | | - * Add to the `volumeMounts` and `volumes` to leverage the map the `logback.xml` file. |
192 | | - * Point to the file in the `REDISCONNECT_LOGBACK_CONFIG` environment variable. |
193 | | -
|
| 1 | +# Redis Connect in K8s |
0 commit comments