Skip to content

Commit 5c37957

Browse files
zzydxmzzydxm
committed
add shard and metadata functionality in pg
1 parent 728c6a0 commit 5c37957

File tree

4 files changed

+1594
-485
lines changed

4 files changed

+1594
-485
lines changed

‎lib/kernel/doc/src/kernel_app.xml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,6 +427,18 @@ MaxT = NetTickTime + NetTickTime / NetTickIntensity</code>
427427
<p>Normally, a terminating node is detected immediately by the transport
428428
protocol (like TCP/IP).</p>
429429
</item>
430+
<tag><marker id="pg_shards_and_metadata"/><c>pg_shards_and_metadata = true | false</c></tag>
431+
<item>
432+
<marker id="pg_shards_and_metadata"></marker>
433+
<p>Enable shards_and_metadata feature for <c>pg</c> server (see
434+
<seeerl marker="pg"><c>pg(3)</c></seeerl>) if the parameter is
435+
<c>true</c>.</p>
436+
<warning><p>Do not enable this feature if any node in the cluster
437+
is from an OTP release prior to OTP 26. <c>pg</c> servers on
438+
such old nodes will crash upon reception of messages in the
439+
extended protocol needed for this feature.</p></warning>
440+
<p>Defaults to <c>false</c>.</p>
441+
</item>
430442
<tag><marker id="prevent_overlapping_partitions"/>
431443
<c>prevent_overlapping_partitions = true | false</c></tag>
432444
<item>

‎lib/kernel/doc/src/pg.xml‎

Lines changed: 246 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@
3737
<module since="OTP 23.0">pg</module>
3838
<modulesummary>Distributed named process groups.</modulesummary>
3939
<description>
40-
<p>This module implements process groups. A message can be sent
41-
to one, some, or all group members.</p>
40+
<p>This module implements a distributed eventual consistent process group
41+
registry.</p>
4242

4343
<p>Up until OTP 17 there used to exist an experimental <c>pg</c>
4444
module in <c>stdlib</c>. This <c>pg</c> module is not the same
@@ -84,6 +84,24 @@
8484
is configured to do so.
8585
</p>
8686

87+
<p>If the <seeapp marker="kernel_app#pg_shards_and_metadata">
88+
<c>shards_and_metadata</c></seeapp> <c>kernel</c> application
89+
parameter is enabled when starting the scope, the server will start with
90+
the <c>shards_and_metadata</c> feature enabled. When enabled
91+
a member on the form <c>{pid(), Metadata :: any()}</c> can also join
92+
or leave the groups.</p>
93+
94+
<p>In addition, when the feature is enabled, besides the singleton group
95+
defined above, the module can also join or leave to a range of shards
96+
for a sharded group. For example, assume we did <c>join([{1, 5}], group,
97+
[Pid1])</c> and then <c>join([{4, 8}], group, [{Pid2, Metadata2}])</c>.
98+
If we do <c>get_metadatas(4, group)</c> it will return <c>[Pid1, {Pid2,
99+
Metadata2}]</c>, and if we do <c>get_metadatas(2, group)</c> it will
100+
return <c>[Pid1]</c>.
101+
This is especially useful for service discovery on sharded services.
102+
Servers can publish the shard ranges that it serves, and client can get
103+
the server pid without explicitly knowing the shard map.</p>
104+
87105
<note><p>
88106
Scope name is used to register process locally, and to name an ETS table.
89107
If there is another process registered under this name, or another ETS table
@@ -93,19 +111,79 @@
93111
<p>A scope can be kept local-only by using a scope name that is unique
94112
cluster-wide, e.g. the node name:</p>
95113
<taglist>
96-
<!-- NOTE THAT THE EMPTY TAG IS INTENTIONAL -->
97-
<tag></tag>
98-
<item><c>pg:start_link(node()).</c></item>
114+
<!-- NOTE THAT THE EMPTY TAG IS INTENTIONAL -->
115+
<tag></tag>
116+
<item><c>pg:start_link(node()).</c></item>
99117
</taglist>
100118
</note>
101119

102120
</description>
103121

104122
<datatypes>
123+
<datatype>
124+
<name name="scope"/>
125+
<desc><p>The identifier of the scope gen_server and ETS table.</p></desc>
126+
</datatype>
105127
<datatype>
106128
<name name="group"/>
107129
<desc><p>The identifier of a process group.</p></desc>
108130
</datatype>
131+
<datatype>
132+
<name name="metadata"/>
133+
<desc><p>The metadata of a process.</p></desc>
134+
</datatype>
135+
<datatype>
136+
<name name="member_metadata"/>
137+
<desc><p>The tuple of a process identifier and its metadata.</p></desc>
138+
</datatype>
139+
<datatype>
140+
<name name="member"/>
141+
<desc><p>A member that can join or leave the groups.</p></desc>
142+
</datatype>
143+
<datatype>
144+
<name name="members"/>
145+
<desc><p>A list of members that can join or leave the groups.</p></desc>
146+
</datatype>
147+
<datatype>
148+
<name name="shard"/>
149+
<desc><p>A non-negative integer shard id.</p></desc>
150+
</datatype>
151+
<datatype>
152+
<name name="shard_range"/>
153+
<desc><p>A shard range (both sides inclusive).</p></desc>
154+
</datatype>
155+
<datatype>
156+
<name name="shard_ranges"/>
157+
<desc><p>A list of shard ranges.</p></desc>
158+
</datatype>
159+
<datatype>
160+
<name name="update"/>
161+
<desc><p>An update request.</p></desc>
162+
</datatype>
163+
<datatype>
164+
<name name="updates"/>
165+
<desc><p>A list of update requests.</p></desc>
166+
</datatype>
167+
<datatype>
168+
<name name="features"/>
169+
<desc><p>A map of enabled features.</p></desc>
170+
</datatype>
171+
<datatype>
172+
<name name="monitor_scope_return_singleton"/>
173+
<desc><p>A monitor scope reply for singleton group.</p></desc>
174+
</datatype>
175+
<datatype>
176+
<name name="monitor_group_return_singleton"/>
177+
<desc><p>A monitor group reply for singleton group.</p></desc>
178+
</datatype>
179+
<datatype>
180+
<name name="monitor_message_singleton"/>
181+
<desc><p>Group membership update message without shards and metadata.</p></desc>
182+
</datatype>
183+
<datatype>
184+
<name name="monitor_message"/>
185+
<desc><p>Group membership update message.</p></desc>
186+
</datatype>
109187
</datatypes>
110188

111189
<funcs>
@@ -133,54 +211,102 @@
133211
<func>
134212
<name name="join" arity="2" since="OTP 23.0"/>
135213
<name name="join" arity="3" since="OTP 23.0"/>
136-
<fsummary>Join a process or a list of processes to a group.</fsummary>
214+
<name name="join" arity="4" since="OTP 27.0"/>
215+
<fsummary>Join a member or a list of members to a group.</fsummary>
137216
<desc>
138-
<p>Joins single process or multiple processes to the
139-
group <c>Group</c>. A process can join a group many times and
217+
<p>If <c>ShardRanges</c> is not specified, joins single member or
218+
multiple members <c>MemberOrMembers</c> to the singleton group
219+
<c>Group</c>. If <c>ShardRanges</c> is specified, joins
220+
<c>MemberOrMembers</c> to the <c>ShardRanges</c> of the sharded
221+
group <c>Group</c>. A member can join a group many times and
140222
must then leave the group the same number of times.</p>
141-
<p><c>PidOrPids</c> may contain the same process multiple times.</p>
223+
<p><c>MemberOrMembers</c> may contain the same member multiple times.
224+
</p>
142225
</desc>
143226
</func>
144227

145228
<func>
146229
<name name="leave" arity="2" since="OTP 23.0"/>
147230
<name name="leave" arity="3" since="OTP 23.0"/>
148-
<fsummary>Make a process leave a group.</fsummary>
231+
<name name="leave" arity="4" since="OTP 27.0"/>
232+
<fsummary>Make a member or a list of members leave a group.</fsummary>
233+
<desc>
234+
<p>If <c>ShardRanges</c> is not specified, makes the member or multiple
235+
members <c>MemberOrMembers</c> leave the singleton group
236+
<c>Group</c>. If <c>ShardRanges</c> is specified, makes
237+
<c>MemberOrMembers</c> leave the <c>ShardRanges</c> of the sharded
238+
group <c>Group</c>. Leaving of a not-joined member will be accepted
239+
as a non-op, leaving a member with only sub-portion of the requested
240+
shard ranges will result in member leaving that sub-portion. However,
241+
if the whole leave request is a non-op, an atom <c>not_joined</c>
242+
will be returned.</p>
243+
</desc>
244+
</func>
245+
246+
<func>
247+
<name name="update" arity="1" since="OTP 27.0"/>
248+
<name name="update" arity="2" since="OTP 27.0"/>
249+
<fsummary>Make updates to a group.</fsummary>
149250
<desc>
150-
<p>Makes the process <c>PidOrPids</c> leave the group <c>Group</c>.
151-
If the process is not a member of the group, <c>not_joined</c> is
152-
returned.</p>
153-
<p>When list of processes is passed as <c>PidOrPids</c>, function
154-
returns <c>not_joined</c> only when all processes of the list
155-
are not joined.</p>
251+
<p>Makes group membership updates, each update is a tuple
252+
<c>update() :: {Group, Position, RemovedMembers, AddedMembers}</c>
253+
where <c>Position</c> is either an atom <c>singleton</c> or a list of
254+
shard ranges, indicating whether this operation is on a singleton
255+
group or specified ranges of a sharded group. <c>AddedMembers</c> is
256+
the list of members to be added, and <c>RemovedMembers</c> is the
257+
list of members to be removed.</p>
258+
<p>The algorithm promises that an <c>update()</c> looks like an
259+
'atomic' operation by a single shard's view if the input shard ranges
260+
are disjoint. It means that a get members query will either return
261+
the expected member list before an update, or after an update. It
262+
won't return with anything else like an intermediate step.</p>
263+
<p>Removing of a not-joined member will be accepted as a non-op.
264+
Removing a member with only sub-portion of the requested shard
265+
ranges will result in member being removed in that sub-portion.
266+
However, if the total update request is a non-op, an atom
267+
<c>not_joined</c> will be returned.</p>
156268
</desc>
157269
</func>
158270

159271
<func>
160272
<name name="monitor_scope" arity="0" since="OTP 25.1"/>
161273
<name name="monitor_scope" arity="1" since="OTP 25.1"/>
274+
<name name="monitor_scope" arity="2" since="OTP 27.0"/>
162275
<fsummary>Starts group membership monitoring for a scope.</fsummary>
163276
<desc>
164277
<p>Subscribes the caller to updates from the specified scope. Returns
165-
content of the entire scope and a reference to match the upcoming
166-
notifications.</p>
278+
content of the entire scope and a reference to match the upcoming
279+
notifications. The content is a map by default; or <c>updates()</c>
280+
if <c>shards_and_metadata</c> is true in <c>Features</c>, the same
281+
as the input in <seemfa marker="#update/2"><c>update/2</c></seemfa>.
282+
</p>
167283

168284
<p>Whenever any group membership changes, an update message is sent
169-
to the subscriber:</p>
285+
to the subscriber. If feature <c>shards_and_metadata</c> is not
286+
enabled, the messages will be:</p>
170287
<code type="none">{Ref, join, Group, [JoinPid1, JoinPid2]}</code>
171288
<code type="none">{Ref, leave, Group, [LeavePid1]}</code>
289+
<p>If feature <c>shards_and_metadata</c> is enabled, the messages will
290+
be:</p>
291+
<code type="none">{Ref, [{Group, Position, RemovedMembers,
292+
AddedMembers}]}</code>
293+
<p>See <seemfa marker="#update/2"><c>update/2</c></seemfa>.</p>
172294
</desc>
173295
</func>
174296

175297
<func>
176298
<name name="monitor" arity="1" since="OTP 25.1"/>
177299
<name name="monitor" arity="2" since="OTP 25.1"/>
300+
<name name="monitor" arity="3" since="OTP 27.0"/>
178301
<fsummary>Starts membership monitoring for a specified group.</fsummary>
179302
<desc>
180303
<p>Subscribes the caller to updates for the specified group. Returns
181-
list of processes currently in the group, and a reference to match
182-
the upcoming notifications.</p>
183-
<p>See <seemfa marker="#monitor_scope/0"><c>monitor_scope/0</c></seemfa>
304+
content currently in the group, and a reference to match the upcoming
305+
notifications. The content is a list of processes by default; or
306+
<c>updates()</c> if <c>shards_and_metadata</c> is true in
307+
<c>Features</c>, the same as the input in
308+
<seemfa marker="#update/2"><c>update/2</c></seemfa>.</p>
309+
<p>See<seemfa marker="#monitor_scope/0"><c>monitor_scope/0</c></seemfa>
184310
for the update message structure.</p>
185311
</desc>
186312
</func>
@@ -197,34 +323,119 @@
197323
</func>
198324

199325
<func>
200-
<name name="get_local_members" arity="1" since="OTP 23.0"/>
201-
<name name="get_local_members" arity="2" since="OTP 23.0"/>
202-
<fsummary>Return all local processes in a group.</fsummary>
326+
<name name="get_all_members" arity="1" since="OTP 23.0"/>
327+
<name name="get_all_members" arity="2" since="OTP 23.0"/>
328+
<name name="get_all_members" arity="3" since="OTP 27.0"/>
329+
<fsummary>Return all members in a group.</fsummary>
203330
<desc>
204-
<p>Returns all processes running on the local node in the
205-
group <c>Group</c>. Processes are returned in no specific order.
206-
This function is optimised for speed.
207-
</p>
331+
<p>Read members in singleton group <c>Group</c>, or <c>Shard</c> of
332+
sharded group <c>Group</c>, depending on if <c>Shard</c> is specified
333+
in the input. Members are returned in no specific order.
334+
This function is optimised for speed.</p>
335+
</desc>
336+
</func>
337+
338+
<func>
339+
<name name="get_all_local_members" arity="1" since="OTP 23.0"/>
340+
<name name="get_all_local_members" arity="2" since="OTP 23.0"/>
341+
<name name="get_all_local_members" arity="3" since="OTP 27.0"/>
342+
<fsummary>Return all local members in a group.</fsummary>
343+
<desc>
344+
<p>Read members in singleton group <c>Group</c>, or <c>Shard</c> of
345+
sharded group <c>Group</c>, depending on if <c>Shard</c> is specified
346+
in the input. Only members running on the local node are returned,
347+
and members are returned in no specific order.
348+
This function is optimised for speed.</p>
208349
</desc>
209350
</func>
210351

211352
<func>
212353
<name name="get_members" arity="1" since="OTP 23.0"/>
213354
<name name="get_members" arity="2" since="OTP 23.0"/>
214-
<fsummary>Return all processes in a group.</fsummary>
355+
<name name="get_members" arity="3" since="OTP 27.0"/>
356+
<fsummary>Return all pid members in a group.</fsummary>
215357
<desc>
216-
<p>Returns all processes in the group <c>Group</c>.
217-
Processes are returned in no specific order.
218-
This function is optimised for speed.</p>
358+
<p>Read members in singleton group <c>Group</c>, or <c>Shard</c> of
359+
sharded group <c>Group</c>, depending on if <c>Shard</c> is specified
360+
in the input. Only <c>pid()</c> type members are returned, and
361+
members are returned in no specific order. This function is for
362+
backward compatibility.</p>
363+
</desc>
364+
</func>
365+
366+
<func>
367+
<name name="get_local_members" arity="1" since="OTP 23.0"/>
368+
<name name="get_local_members" arity="2" since="OTP 23.0"/>
369+
<name name="get_local_members" arity="3" since="OTP 27.0"/>
370+
<fsummary>Return all local pid members in a group.</fsummary>
371+
<desc>
372+
<p>Read members in singleton group <c>Group</c>, or <c>Shard</c> of
373+
sharded group <c>Group</c>, depending on if <c>Shard</c> is specified
374+
in the input. Only <c>pid()</c> type members which are also running
375+
on the local node are returned, and members are returned in no
376+
specific order. This function is for backward compatibility.</p>
219377
</desc>
220378
</func>
221379

222380
<func>
223381
<name name="which_groups" arity="0" since="OTP 23.0"/>
224382
<name name="which_groups" arity="1" since="OTP 23.0"/>
225-
<fsummary>Return a list of all known groups.</fsummary>
383+
<fsummary>Returns a list of all known singleton groups.</fsummary>
384+
<desc>
385+
<p>Returns a list of all known singleton groups.</p>
386+
</desc>
387+
</func>
388+
389+
<func>
390+
<name name="which_local_groups" arity="0" since="OTP 23.0"/>
391+
<name name="which_local_groups" arity="1" since="OTP 23.0"/>
392+
<fsummary>Returns a list of singleton groups that have any local member
393+
joined.</fsummary>
394+
<desc>
395+
<p>Returns a list of singleton groups that have any local member
396+
joined.</p>
397+
</desc>
398+
</func>
399+
400+
<func>
401+
<name name="which_sharded_groups" arity="0" since="OTP 27.0"/>
402+
<name name="which_sharded_groups" arity="1" since="OTP 27.0"/>
403+
<fsummary>Returns a list of all known sharded groups.
404+
</fsummary>
405+
<desc>
406+
<p>Returns a list of all known sharded groups.</p>
407+
</desc>
408+
</func>
409+
410+
<func>
411+
<name name="which_sharded_local_groups" arity="0" since="OTP 27.0"/>
412+
<name name="which_sharded_local_groups" arity="1" since="OTP 27.0"/>
413+
<fsummary>Returns a list of all known sharded groups that have any local
414+
member joined</fsummary>
415+
<desc>
416+
<p>Returns a list of all known sharded groups that have any local
417+
member joined.</p>
418+
</desc>
419+
</func>
420+
421+
<func>
422+
<name name="which_shard_ranges" arity="1" since="OTP 27.0"/>
423+
<name name="which_shard_ranges" arity="2" since="OTP 27.0"/>
424+
<fsummary>Returns a list of all known shard ranges of a sharded group.
425+
</fsummary>
426+
<desc>
427+
<p>Returns a list of all known shard ranges of a sharded group.</p>
428+
</desc>
429+
</func>
430+
431+
<func>
432+
<name name="which_local_shard_ranges" arity="1" since="OTP 27.0"/>
433+
<name name="which_local_shard_ranges" arity="2" since="OTP 27.0"/>
434+
<fsummary>Returns a list of shard ranges of a sharded group that have
435+
any local member joined.</fsummary>
226436
<desc>
227-
<p>Returns a list of all known groups.</p>
437+
<p>Returns a list of shard ranges of a sharded group that have any
438+
local member joined.</p>
228439
</desc>
229440
</func>
230441

0 commit comments

Comments
 (0)