@@ -96,84 +96,84 @@ The project requires a running Cassandra database instance and a Redis database.
96
96
Call this under the project folder setup them on your local environment.
97
97
98
98
``` sh
99
- docker-compose up -d
99
+ docker-compose up -d
100
100
```
101
101
Cassandra needs a bit of time to establish its configuration.
102
102
The following command will help to find out if it is ready.
103
+
103
104
``` sh
104
- docker exec -it cassandra-service cqlsh -e " describe keyspaces"
105
- ```
105
+ docker exec -it cassandra-service cqlsh -e " describe keyspaces"
106
+ ```
107
+
106
108
Our database is ready, if there is no error.
107
109
The following line will create a keyspace and table on the database.
108
110
109
111
``` sh
110
- sudo sh ./scripts/provision.sh
112
+ sudo sh ./scripts/provision.sh
111
113
```
112
114
113
115
Now the setup is ready for our projects. Let's run the ` job ` command first. This
114
116
command pulls all the product files from AWS, processes them via a pipeline, and
115
117
stores them in a database. The first run might take a bit longer since we don't
116
- profit from the cache at the first run.
118
+ profit from the cache for the first run.
117
119
118
120
``` sh
119
121
go run cmd/job/main.go -config dataflow.conf
120
122
```
121
123
122
- For example, if we try the following command, we'll get a slower duration of execution.
124
+ If we try the following command, we'll get a slower duration of execution.
123
125
``` sh
124
- go run cmd/job/main.go -config dataflow.conf -concurrency 1
126
+ go run cmd/job/main.go -config dataflow.conf -concurrency 1
125
127
```
126
128
127
129
Start the ` microservice ` HTTP daemon.
128
130
``` sh
129
131
130
- go run cmd/microservice/main.go -config dataflow.conf
132
+ go run cmd/microservice/main.go -config dataflow.conf
131
133
```
132
134
133
135
Test it:
134
136
``` sh
135
137
136
- curl localhost:8080/product/42
138
+ curl localhost:8080/product/42
137
139
```
138
140
139
141
### Using ` redis ` cache
140
142
To look into caching via redis, we can do a demonstration.
141
143
First on the project directory call the following
142
144
command to remove the database instances.
143
- ``` sh
144
- docker-compose down --remove-orphans
145
+ ``` sh
146
+ docker-compose down --remove-orphans
145
147
146
148
```
147
149
Then re-create fresh instances of Cassandra and Redis..
148
- ```
149
- sh docker-compose up -d
150
+ ``` sh
151
+ docker-compose up -d
150
152
```
151
153
Wait like 30 seconds to let Cassandra stand up.
152
154
Use this as a readiness probe:
153
155
``` sh
154
-
155
- docker exec -it cassandra-service cqlsh -e " describe keyspaces"
156
+ docker exec -it cassandra-service cqlsh -e " describe keyspaces"
156
157
```
157
158
158
159
After this, we have a pipeline setup with an empty cache layer.
159
160
This means the data processing should take a bit longer. Let's check.
160
161
This command measures how much time elapsed for the execution of the job.
161
162
162
163
``` sh
163
- go run cmd/job/main.go -config dataflow.conf
164
+ go run cmd/job/main.go -config dataflow.conf
164
165
```
165
166
I got the following result on my computer.
166
167
167
168
``` sh
168
- Pipeline tooks 29.788467364s
169
+ Pipeline tooks 29.788467364s
169
170
```
170
171
171
172
After the second execution of the same command, I got the following output on my local.
172
173
``` sh
173
-
174
- Pipeline tooks 15.399257129s
174
+ Pipeline tooks 15.399257129s
175
175
```
176
- This result shows the efficiency of the optimization through caching.
176
+ This result shows benefit of caching.
177
177
We save half of the execution time of the pipeline.
178
178
179
179
0 commit comments