Docker image: the background join has no server role, so a standalone replication server or a directory server without one needs the deprecated one-shot types #1178

(line numbers on master, 3f4deb9)

Problem

Since #1086 (#1115) the image joins its replication topology in the background, on every start, with OPENDJ_REPLICATION_TYPE=simple, and the README declares the one-shot types srs, sdsr and rg deprecated in favour of it (README.md:182-189). But simple builds one kind of topology only: every server is a directory server and a replication server at once. The two other roles dsreplication enable offers - a replication server that holds no data (--onlyReplicationServerN) and a directory server that connects to replication servers elsewhere (--noReplicationServerN) - are reachable through the deprecated types alone. The deprecation leaves no supported way to run them.

That layout is the one of #534 (four directory servers without a replication server, two standalone replication servers - the topology that piled up missing changes under load), the one #1090 wants to test, and the usual production one, where the replication servers are kept off the servers that take the client load.

What simple does

• One enable, both sides combined. The join runs a single dsreplication enable (join.sh:442-449) with --replicationPort1 and --replicationPort2 and neither role flag. Without them the tool configures both a replication server and a replication domain on each side (ReplicationCliArgumentParser.java:100-108).
• The peer's role is not this server's to choose, and the enable chooses it anyway. A server without a replication server that is passed through without --noReplicationServer1 is given one: !serverDesc.isReplicationServer() && enableServer.configureReplicationServer() runs configureAsReplicationServer() (ReplicationCliMain.java:5087-5097). A role variable for the joining server alone is therefore not enough: as the call stands, every directory server without a replication server that a new member joins through silently becomes a combined one. Likewise, a replication-server-only peer passed without --onlyReplicationServer1 is asked to configure a domain for BASE_DN (ReplicationCliMain.java:5059-5060).
• Membership means holding the data. A member holds the replication domain of BASE_DN and is registered in cn=admin data (replicates_base_dn(), join.sh:302-309). A replication server without data never has that domain, so the join cannot recognise one as a member.
• Health waits for the data. A volume bootstrapped with simple carries INITIALIZE_PENDING until the join has initialized it from the topology (run.sh:192-194), and the health marker waits for the join (run.sh:156, run.sh:229-236). A replication server has no BASE_DN data to initialize.
• The seed is a data holder. The first peer of REPLICATION_PEERS seeds a topology nobody else holds with its own data. A replication server has none to seed with.
• The bootstrap always creates the data backend. setup.sh creates userRoot for BASE_DN on every server (setup.sh:86-88), and with ADD_BASE_ENTRY / SAMPLE_DATA fills it (:90-104). On a replication server that would be data nothing replicates, served to any client that reaches it.

What the deprecated types do

replicate.sh:87-150:

• srs enables this server as a directory server without a replication server (--noReplicationServer1) and MASTER_SERVER as a replication server only (--onlyReplicationServer2), then runs initialize-all. The role of the master is set by whichever replica enables first, not by the master. Every replica pairs with the one MASTER_SERVER, so nothing connects a second replication server: the two standalone replication servers of Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 cannot be built this way.
• sdsr enables this server as a directory server without a replication server next to a combined master, and initializes it from that master.
• Both run once, during the first bootstrap, against a fixed MASTER_SERVER, with none of the retries, membership repair and health gating Docker image: joining replication is one-shot, fixed-master and unchecked, which a StatefulSet cannot rely on #1086 brought to simple.
• CI covers sdsr (docker-test-replication.sh:617-618); srs and rg are run by no test.

Proposal

A server role for simple, say REPLICATION_ROLE (the name is open):

1. Both sides of every enable get their actual role. This server's comes from its environment. The peer's is read from the peer - whether its configuration holds a replication server and a domain for BASE_DN - and never assumed, so a peer without a replication server keeps having none.
2. Membership by role. A replication member has its replication server and is registered in cn=admin data (which the enable replicates to a replication-server-only server as well, ReplicationCliMain.java:5059-5060). A directory member has its BASE_DN domain, is registered, and its domain lists at least one replication server.
3. Bootstrap by role. replication creates no userRoot and ignores ADD_BASE_ENTRY / SAMPLE_DATA, never carries INITIALIZE_PENDING, never seeds and is never initialized. Its health follows its membership.
4. Seeding. Only a server that holds data (combined or directory) seeds. A topology needs at least one replication server: a directory server whose peers hold none does not join, and says so rather than retrying for ever.
5. Docs. README: the roles, the Kubernetes layout (one StatefulSet per role, REPLICATION_PEERS listing the members of both), and how srs / sdsr map onto simple with roles, so that their deprecation has a replacement.
6. Tests in docker-test-replication.sh: two replication + two directory servers, a change on each directory server reaches the other; a restart of each; a new member joined through a directory peer leaves that peer without a replication server; a replication server holds no BASE_DN backend; a directory server with no replication server among its peers does not turn healthy.

Out of scope

• rg (replication groups, OPENDJ_REPLICATION_GROUP_ID) has the same gap - deprecated with no simple counterpart - but is a separate setting.
• The Helm chart (Add an official Helm chart so OpenDJ can be deployed on any Kubernetes cluster #1079), which would use the roles.

Related

• Docker image: joining replication is one-shot, fixed-master and unchecked, which a StatefulSet cannot rely on #1086 / [#1086] Join replication in the background on every start of the Docker image #1115 - the background join this extends.
• Replication: no test writes to several replicas under load and then checks that they hold the same data #1090 - the convergence test needs the Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 topology. Until this lands it builds it with explicit dsreplication enable --noReplicationServerN / --onlyReplicationServerN through docker exec, on containers started without OPENDJ_REPLICATION_TYPE.
• Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 - the topology.
• OpenShift template: replication parameters are never passed, the Secret is unused and malformed, and nothing tests it #1089 - the OpenShift template, whose replication parameters (OPENDJ_REPLICATION_TYPE, OPENDJ_MASTER_SERVER) predate simple.