-
Notifications
You must be signed in to change notification settings - Fork 0
/
notes.txt
169 lines (121 loc) · 7.06 KB
/
notes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
Nats
https://nats.io/
Nats JS(jetstream)
https://docs.nats.io/nats-concepts/jetstream
https://docs.nats.io/using-nats/developer/develop_jetstream
https://docs.nats.io/nats-concepts/core-nats/queue#stream-as-a-queue
JetStream wire API Reference
https://docs.nats.io/reference/reference-protocols/nats_api_reference
Shemas
https://github.com/nats-io/jsm.go/tree/main/schema_source/jetstream/api/v1
JetStream Model Deep Dive
https://docs.nats.io/using-nats/developer/develop_jetstream/model_deep_dive
Pedantic mode
https://docs.nats.io/using-nats/developer/connecting/misc#turn-on-pedantic-mode
Go JetStream Simplified Client
https://github.com/nats-io/nats.go/tree/main/jetstream#jetstream-simplified-client
Nats JS client for php
https://github.com/basis-company/nats.php
Description of usage
https://github.com/basis-company/nats.php/issues/59#issuecomment-1907919039
Ephemeral Consumer life cycle
https://natsio.slack.com/archives/CM3T6T7JQ/p1711275996676309
https://natsio.slack.com/archives/CM3T6T7JQ/p1711278584957209?thread_ts=1711275996.676309&cid=CM3T6T7JQ
Nats articles
https://www.byronruth.com/tag/series/
NATS and Docker
https://docs.nats.io/running-a-nats-service/nats_docker
RoadRunner Jobs
https://docs.roadrunner.dev/queues-and-jobs/overview-queues
Nats Jobs
https://docs.roadrunner.dev/queues-and-jobs/nats
https://github.com/roadrunner-server/nats/tree/master/natsjobs
Beanstalk Jobs
https://docs.roadrunner.dev/queues-and-jobs/beanstalk
Allowed number of consumers for JS as Queue (WorkQueuePolicy)
https://natsio.slack.com/archives/CM3T6T7JQ/p1712153925255479
Multiple clients/workers as one consumer
https://natsio.slack.com/archives/CM3T6T7JQ/p1714853594923729?thread_ts=1714853415.006099&cid=CM3T6T7JQ
Synadia examples
https://github.com/synadia-io/rethink_connectivity
Implementing NATS JetStream as Queues
https://dzone.com/articles/implementing-nats-jetstream-as-queues-in-golang-a
Choria Asynchronous Job Processor
https://github.com/choria-io/asyncjobs
NATS blog
https://nats.io/blog/
JetStream Java tutorial
https://nats.io/blog/hello-world-java-client/
JetStream stream creation in Java
https://nats.io/blog/jetstream-java-client-01-stream-create/
JetStream publishing in Java
https://nats.io/blog/jetstream-java-client-02-publish/
Consumers in Java
https://nats.io/blog/jetstream-java-client-03-consume/
Push consumers in Java
https://nats.io/blog/jetstream-java-client-04-push-subscribe/#jetstream-push-consumers-with-the-natsio-java-library
Pull consumers in Java
https://nats.io/blog/jetstream-java-client-05-pull-subscribe/#jetstream-pull-consumers-with-the-natsio-java-library
[Questions]Regarding Adapter implementation
https://github.com/yiisoft/queue-redis/issues/4
PHP UUID generator (for usage as MESSAGE_ID_KEY)
https://github.com/ramsey/uuid
Tracking of job state in Work-queue stream
https://natsio.slack.com/archives/CM3T6T7JQ/p1715060131243269
Question:
Tracking of job state in Work-queue stream
----------------------------------------------------
Proposed flow:
Jobs are submitted to JS work-queue stream
Received sequence number is used further as unique JOB ID
Job has following states:
Submitted
In-process
Finished
Failed
Submitter calls GetJobState(JOB ID) in order to get job state
Worker calls SetJobState(JOB ID, job state) in order to update job state
This is typical flow in background job processing.
What is the NATS way/pattern to implement it?
Answer:
I’m using a JetStream configured as a workflow for sending job updates
and a K/V store for saving the state of the jobs in JSON format.
I Ack() the job after updating the K/V. That is hopefully good enough.
Nats lock
https://pkg.go.dev/github.com/metal-toolbox/addonx/natslock
JetStream Global Lock
https://natsio.slack.com/archives/CM3T6T7JQ/p1715835870966629
Question:
For background processing we use:
Stream with WorkQueuePolicy - queue for workers
KV bucket for saving of job states (submitted/inprocess/finished/failed)
Is there possibility to lock JetStream in order to achieve atomic working with 2 storages?
Another example when atomic is required - initiation of streams/buckets.
We can not do it via cli, so first process (submitter or worker) should create/purge/etc
by itself
Answers:
... the solution is simple, optimistic locking with deduplication:
Read from stream (but no ack)
CAS (compare and swap) Store to KV (only update if a unique transaction key is not already set)
If update successful - ack in stream
If update failed (the transaction key is already in KV) its a duplicate. You log a warning and ack the message anyway
...You dont generally need a global lock for this kind of thing since jetstream
ensures only 1 client gets a message within the ack waiting time.
So optimistic concurrency control is often enough when you consider that
if you use acks correctly and request more time to process a message if you are
not done with it in time you can even handle long running jobs this way
Run GitActions locally
https://nektosact.com/
Repo for testing
https://github.com/cplee/github-actions-demo
Check streams
cd ./bin/amd64
./natscli s ls -a
╭─────────────────────────────────────────────────────────────────────────────────────────╮
│ Streams │
├──────────────────┬─────────────┬─────────────────────┬──────────┬────────┬──────────────┤
│ Name │ Description │ Created │ Messages │ Size │ Last Message │
├──────────────────┼─────────────┼─────────────────────┼──────────┼────────┼──────────────┤
│ YII-QUEUEJOBS │ │ 2024-05-26 14:50:48 │ 0 │ 0 B │ never │
│ KV_YII-QUEUEJOBS │ │ 2024-05-26 14:50:49 │ 468 │ 26 KiB │ 17ms │
╰──────────────────┴─────────────┴─────────────────────┴──────────┴────────┴──────────────╯