dedicated event etcd draft PoC for ShiftWeek

[tjungblu]

/hold

just here for CI runs and cluster bot builds

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

• do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• 🔍 Trigger a full review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [tjungblu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tjungblu]

Some quick benchmark results using:

etcdctl check perf --load="xl"

This is running against the normal etcd:

export ETCDCTL_ENDPOINTS="https://localhost:2379"
etcdctl check perf --load="xl" 
FAIL: Throughput too low: 8097 writes/s
PASS: Slowest request took 0.077356s
PASS: Stddev is 0.004320s
FAIL

This is running against localhost in-memory:

export ETCDCTL_ENDPOINTS="https://localhost:20379"
etcdctl check perf --load="xl"
... 
FAIL: Throughput too low: 11476 writes/s
PASS: Slowest request took 0.068861s
PASS: Stddev is 0.002417s

Now this is pretty crappy, let's try some tuning:

unsafe-no-fsync=true

FAIL: Throughput too low: 11407 writes/s
PASS: Slowest request took 0.061991s
PASS: Stddev is 0.001785s

seems has no effect

--backend-batch-limit=50000 and --backend-batch-interval=1m up from 10000 / 100ms

FAIL: Throughput too low: 11438 writes/s
PASS: Slowest request took 0.082044s
PASS: Stddev is 0.002382s

also no real effect, leading me to believe this benchmark is not really disk bound in the first place.

Checking the allocation route with:

--backend-bbolt-freelist-type=array instead of the hashmap implementation yields

FAIL: Throughput too low: 11630 writes/s
PASS: Slowest request took 0.066573s
PASS: Stddev is 0.001428s

Performs only slightly better.

---

After CPU profiling, some interesting findings:

• most of the time is spent on netFD.write syscall, flushing the response of etcd to the network
• there is significant contention in the .Put and processInternalRaftRequestOnce call chain,
• fsync just is 0.9%, also matching what we saw above
• the apply loop is ~6%, matching the results when changing the batch settings

Unfortunately the grpc options for buffer sizes (R+W are at 32K each) need recompilation, so I won't be able to max this out any further today.

[tjungblu]

this requires dnsPolicy=ClusterFirstWithHostNet on the kube-apiserver static pods

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale