Code

Kafka perf improvements#1964

Code

Merged

slinkydeveloper merged 6 commits intorestatedev:mainrestatedev/restate:mainfrom slinkydeveloper:kafka-improvementsslinkydeveloper/restate:kafka-improvementsCopy head branch name to clipboardSep 18, 2024

+137-25

Commits

Commits on Sep 17, 2024

• Use topic partition queue splitting feature from rdkafka. This allows to split the main consumer queue into subqueues, such that we can spawn a subtask for each topic-partition tuple.

  Show description for 3819398
  slinkydeveloper

  committed
  3819398

  Copy full SHA for 3819398

• Bump to rust-rdkafka 0.35, the latest release that seems to be ok. See fede1024/rust-rdkafka#638 (comment) for more details.

  Show description for e2664cb
  slinkydeveloper

  committed
  e2664cb

  Copy full SHA for e2664cb

• Add restate.kafka_ingress.requests.total metric. This is useful to graph the consumption rate.

  slinkydeveloper

  committed
  bbdb7cd

  Copy full SHA for bbdb7cd

• Feedback

  slinkydeveloper

  committed
  8762e9d

  Copy full SHA for 8762e9d

• Plumb task center from constructor

  slinkydeveloper

  committed
  805a43b

  Copy full SHA for 805a43b

• Make sure we cancel child tasks when we're shutting down

  slinkydeveloper

  committed
  778dd08

  Copy full SHA for 778dd08