Skip to content

[#1144] Widen the ephemeral port range before the benchmark runs - #1145

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issue-1144-benchmark-ephemeral-ports
Oct 1, 2026
Merged

vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issue-1144-benchmark-ephemeral-ports

Conversation

@vharseko

@vharseko vharseko commented Oct 1, 2026

Copy link
Copy Markdown
Member

Fixes #1144

Problem

The BIND sampler of the LDAP benchmark opens a new connection on every iteration, about 500 per second at the current CI throughput. With -p 1389:1389, docker-proxy relays each connection to the container over a second connection from an ephemeral port. With 60 s of TIME_WAIT, that leg outgrows the default range (32768–60999, 28 232 ports) within about a minute. The proxy then drops new connections, and BIND fails with LDAP connection has been closed: 1911 and 1047 failures in run 36776405938.

Change

Before the benchmarks run, set:

  • net.ipv4.ip_local_port_range="1024 65535", which roughly doubles the number of ephemeral ports;
  • net.ipv4.tcp_tw_reuse=1, which lets connect() reuse ports held in TIME_WAIT.

The settings go in:

  • .github/benchmark/compare-opendj.sh, used by all three benchmarks in build.yml (je vs pdb and Build vs Release in build-docker, Build vs Release in build-docker-alpine);
  • .github/workflows/benchmark.yml (OpenLDAP vs OpenDJ), as a separate step before the servers start.

Verification

  • bash -n on the script and a YAML parse of the workflow both pass.
  • The benchmarks of this PR's own Build run exercise compare-opendj.sh. benchmark.yml runs only on workflow_dispatch or after Release.
  • One green run does not prove the fix: below ~3600 tests/s the failures did not appear before either. The port-exhaustion explanation is inferred from the throughput numbers; the TIME_WAIT count was not measured.

… benchmark runs

BIND opens a new connection per iteration, and docker-proxy relays each one to
the container over a second connection from an ephemeral port. At ~500 binds/s
and 60 s of TIME_WAIT that leg outgrows the default range (32768-60999) within a
minute; the proxy then drops new connections and BIND fails with "LDAP
connection has been closed".

Before the benchmarks of compare-opendj.sh and of the OpenLDAP vs OpenDJ
workflow run, set net.ipv4.ip_local_port_range to 1024-65535 and enable
net.ipv4.tcp_tw_reuse.

Fixes OpenIdentityPlatform#1144

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The setting goes in before any server starts, on both benchmark paths.

  • .github/benchmark/compare-opendj.sh:59 sits after the dependency block and before the first bench_one, so all three build.yml benchmarks (je vs pdb, Build vs Release in build-docker and build-docker-alpine) run with it.
  • .github/workflows/benchmark.yml:97-104 runs the step before "Start OpenLDAP", so OpenLDAP and OpenDJ are both measured with the same settings. Ports 1389/2389 now fall inside the range, but connect() does not take them: Linux scans ports of the low bound's parity first and skips ports that are already bound.

@vharseko
vharseko merged commit ee4d713 into OpenIdentityPlatform:master Oct 1, 2026
24 checks passed
@vharseko
vharseko deleted the issue-1144-benchmark-ephemeral-ports branch October 1, 2026 13:42
vharseko added a commit that referenced this pull request Oct 2, 2026
…ose message holds a comma reaches the step log (#1151)

Fixes #1150

### Problem

`compare-opendj.sh` listed the errors of each benchmark side with `awk
-F','`. JMeter quotes a `responseMessage`
that contains a comma, and every message naming a DN does, so the
columns after it shifted and such rows were
never matched as failures. On the PDB side this hid the result-80 and
result-32 rows of #1149 and left only
`BIND | 49`, the last consequence of a failed ADD.

### Change

The error listing in `bench_one` now reads the JTL with `python3`'s
`csv` module, which is already used by the
same Docker jobs in `build.yml`.

- Failures are grouped by operation, result code and message, with
digits in the message replaced by `N`. The
messages carry the entry DN or the elapsed milliseconds, and without
this each failed row would be a kind of its
  own.
- The step log shows the number of failed rows and kinds of error, the
ten most frequent kinds and, when there are
  more, how many are left out.
- The listing still goes to stderr, because stdout of `bench_one`
carries the server version. `|| true` keeps a
  failure of the listing from stopping the benchmark.

The `err` columns of the step summary come from `statistics.json`
through `summary.sh` and are unchanged.

### Verification

The block, cut verbatim from the script, run on the
`benchmark-pdb-vs-je` artifacts named in the issue:

| JTL | Before (awk) | After |
|---|---|---|
|
[36845050319](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/36845050319),
PDB | `2 BIND \| 49` | 12 failed rows, 6 kinds of error:
DELETE/ADD/READD 80, COMPARE/MODIFY 32, BIND 49 |
|
[36738515054](https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/36738515054),
PDB | `5131 BIND \| 800`, `2 BIND \| 49` | 5145 failed rows, 7 kinds of
error, with the 80 and 32 rows present |
| JE side and both build-vs-release runs | nothing | nothing (no failed
rows) |

Output for run 36845050319, PDB:

```
[b] 12 failed rows, 6 kinds of error (count | op | code | message):
      3 DELETE | 80 | pdb: backend 'userRoot' did not apply the transaction after N attempts in N ms, the attempt cap being spent; the last conflict was com.persistit.exception.RollbackException
      2 ADD | 80 | pdb: backend 'userRoot' did not apply the transaction after N attempts in N ms, the attempt cap being spent; the last conflict was com.persistit.exception.RollbackException
      2 COMPARE | 32 | The specified entry mail=u_N_N@test.com,ou=People,dc=example,dc=com does not exist in the Directory Server
      2 MODIFY | 32 | Entry mail=u_N_N@test.com,ou=People,dc=example,dc=com cannot be modified because no such entry exists in the server
      2 BIND | 49 | Invalid Credentials
      1 READD | 80 | pdb: backend 'userRoot' did not apply the transaction after N attempts in N ms, the attempt cap being spent; the last conflict was com.persistit.exception.RollbackException
```

A synthetic JTL with 13 kinds of error, messages holding commas and one
message spanning two lines printed ten
kinds and `... 3 more kinds not shown`. When the JTL is missing or
malformed, the block exits with 0. In every case
nothing was written to stdout. `bash -n` passes.

#1145 edits the same file in a different place (the `sysctl` near the
top), so the two do not conflict.
vharseko added a commit that referenced this pull request Oct 2, 2026
…nresolved (#1148)

Fixes #1147

### Problem

A page that uses an attribute it does not define, or a verbatim block
that uses an attribute without attribute subs, publishes `{name}`
literally, and the build stays green. The only render in the build is
the PDF book, where an attribute defined in an earlier chapter also
resolves in the later ones; the site renders every chapter as a page of
its own, so a page can be broken on the site while the PDF is fine
(`chap-uninstall.adoc` before #1140). The `antora` goal of
`doc-maven-plugin` only rewrites the `.adoc` files, it renders nothing.

### Change

- New `check-attribute-references` execution of
`asciidoctor-maven-plugin` in the `man-pages` profile (phase `verify`,
so it runs on the ubuntu legs of the PR build). It renders every `.adoc`
of `target/asciidoc/source` on its own to throwaway HTML, with each
file's own directory as the base, and fails on any WARN that mentions an
attribute:
  1. an attribute that is not defined: `attribute-missing=warn`;
2. an attribute in a listing, literal or passthrough block without
attribute subs, or in a literal table cell (`l|`, `cols="1l"`): the new
`literal-attribute-references.rb` tree processor. A reference counts
when its name is an attribute at that point of the page, an intrinsic
one such as `{nbsp}`, or one that a `:name:` entry anywhere in the doc
sources sets - the last catches a page that neither defines nor
substitutes the attribute, and since the page is among those sources,
its own entry counts even where it has unset the attribute. Braces
around any other name (`{SSHA}` values, `{cn}` in MakeLDIF templates)
are left alone. Without attribute subs a backslash does not escape, so
`\{name}` is reported too, as published with its backslash.
The advice in the warning is `subs="+attributes"`, which keeps the
block's default subs; a literal cell takes no subs, so it is told to
become an `a|` cell with such a listing.
- `LiteralAttributeReferencesTest` reads the execution's configuration
from the pom - the required extensions, the attributes, the `failIf` -
renders pages with it, and checks which of them would fail the build: 13
that must fail, 9 that must pass. Its sources directory sits under
`target` and holds one of the pages, as the build's does, and it asserts
that `sourceDirectory` and `literal-attribute-sources` are the same
directory. The pom module binds `testCompile` and `test` for it, and the
plugin and the test share `asciidoctorj` `${asciidoctorj.version}`
(2.5.11, the plugin's own).
- The 16 blocks of the guides with `subs="attributes"` take
`subs="+attributes"`. The replacing form had dropped
`specialcharacters`, so `chap-writing-plugins` published `(build
<unknown>, revision <unknown>)` as raw tags, which a browser hides; the
`>`, `>>>>` and `&&` of the other blocks were raw as well. The rendered
HTML of the six pages differs only in that escaping.
- The two literal placeholders the check found on master are escaped
like the generated reference does since #1142: `{options}` in the
`windows-service` synopsis and `{path}` in the password policy chapter
(both were shown with braces, but by accident).

Asciidoctor resets the document attributes to the header before tree
processors run, so the extension replays the attribute entries of the
body in document order, as the converter does, on a copy of each
document's attributes. The document itself is left alone: playing the
entries back on it would also hand the compat mode of the last
`compat-mode` entry to the converter.

### Limits

- `asciidoctor-maven-plugin` 2.2.6 evaluates `failIf` after each file,
so the build stops at the first page that fails; fix it and run `mvn
-Pdistribution-unix,man-pages -pl opendj-doc-generated-ref
asciidoctor:process-asciidoc@check-attribute-references` again for the
next one (both profiles activate by themselves only on Linux). This is
noted in the pom.
- Asciidoctor names no file for a missing attribute; it is the page of
the `Converted` line that follows the warning.
- The plugin makes every Maven project property an attribute
(`product.name` is `{product-name}`), and the site defines none of them,
so a page that uses one passes the check and still shows the braces on
the site. The plugin has no switch for it; no page uses one today. This
is noted in the pom.
- The `partials` directory is rendered as well (234 of the 512 files),
although Antora only includes its files and never publishes them as
pages; the plugin has no exclusion, and an `_` prefix would break the
includes. So the man pages are checked twice, and a partial that relies
on an attribute of the chapter that includes it would fail here although
the site resolves it; none of the 11 partials that exist only there
holds an attribute reference. This is noted in the pom.

### Verification

- Current master (with #1145 and #1146), with the extension of this
round: 512 pages, no warning at all, `BUILD SUCCESS`; the check itself
took 43 s in the CI job of the previous round. The hand-written pages
were pre-processed again from this branch; the generated reference pages
come from my last module build.
- Before #1140 it reported exactly what #1140 fixed: the undefined
`{opendj-version}` in `chap-uninstall.adoc` and the listing without subs
in `chap-monitoring.adoc`.
- Three mutants on the pre-processed copies each fail the build:
`:opendj-version:` removed from `chap-uninstall`; `subs="attributes"`
removed from a listing in `chap-monitoring`; both the subs and the
definition removed from `chap-monitoring` (caught only through the
`:name:` entries of the other pages).
- `LiteralAttributeReferencesTest`: 22 green. Each of 16 mutants turns
it red - in the extension: no report at all, the subs guard inverted, no
literal-cell branch, no intrinsic names, no replay of body entries, no
unset in the replay, the replay on the document instead of a copy, the
sources path back in the glob pattern, `\{name}` skipped, the old
`subs="attributes"` advice; in the pom: a typo in `containsText`,
`failIf` on `ERROR`, no `attribute-missing`, no
`literal-attribute-sources`, the extension not required,
`sourceDirectory` narrowed to `man-pages`. The previous round's
extension, which played the entries back on the document, fails the two
compat-mode rows.
- The HTML rendered with and without the extension is identical on all
512 pages (only the `Last updated` line set aside), so the extension
changes nothing in the output.
- Edge cases checked by hand: header and body attributes, unset
attributes, compat mode set or unset in the body, literal paragraphs,
passthrough blocks, listings inside AsciiDoc table cells, literal cells,
`{SSHA}`/`{givenName}` braces, a sources path with `{` and `[`.
- Not run locally: the full module build with the PDF (killed for memory
on my machine); the check ran on the sources the build had already
pre-processed. CI runs the whole path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark BIND fails with "LDAP connection has been closed" once docker-proxy runs out of ephemeral ports

2 participants