kafka-proxy

Work in progress

The Kafka Proxy is based on idea of Cloud SQL Proxy. It allows a service to connect to Kafka brokers without having to deal with SASL/PLAIN authentication and SSL certificates.

It works by opening tcp sockets on the local machine and proxying connections to the associated Kafka brokers when the sockets are used. The host and port in Metadata and FindCoordinator responses received from the brokers are replaced by local counterparts. For discovered brokers (not configured as the boostrap servers), local listeners are started on random ports. The dynamic local listeners feature can be disabled and an additional list of external server mappings can be provided.

The Proxy can terminate TLS traffic and authenticate users using SASL/PLAIN. The credentials verification method is configurable and uses golang plugin system over RPC.

Kafka API calls can be restricted to prevent some operations e.g. topic deletion.

See:

• A Guide To The Kafka Protocol
• Kafka protocol guide

Building

make build.docker-build

Usage example

build/kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,0.0.0.0:32399"

build/kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" \
                         --bootstrap-server-mapping "192.168.99.100:32401,127.0.0.1:32401" \
                         --bootstrap-server-mapping "192.168.99.100:32402,127.0.0.1:32402" \
                         --dynamic-listeners-disable

build/kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" \
                         --external-server-mapping "192.168.99.100:32401,127.0.0.1:32402" \
                         --external-server-mapping "192.168.99.100:32402,127.0.0.1:32403" \
                         --forbidden-api-keys 20

build/kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9093,0.0.0.0:32399" \
                         --tls-enable --tls-insecure-skip-verify \
                         --sasl-enable -sasl-username myuser --sasl-password mysecret

Proxy authentication example

make clean build plugin.auth-user && build/kafka-proxy server --proxy-listener-key-file "server-key.pem"  \
                         --proxy-listener-cert-file "server-cert.pem" \
                         --proxy-listener-ca-chain-cert-file "ca.pem" \
                         --proxy-listener-tls-enable \
                         --proxy-listener-auth-enable \
                         --proxy-listener-auth-command build/auth-user \
                         --proxy-listener-auth-param "--username=my-test-user" \
                         --proxy-listener-auth-param "--password=my-test-password"

make clean build plugin.auth-ldap && build/kafka-proxy server \
                         --proxy-listener-auth-enable \
                         --proxy-listener-auth-command build/auth-ldap \
                         --proxy-listener-auth-param "--url=ldaps://ldap.example.com:636" \
                         --proxy-listener-auth-param "--user-dn=cn=users,dc=exemple,dc=com" \
                         --proxy-listener-auth-param "--user-attr=uid" \
                         --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"

What should be done

• Metadata response versions V0,V1,V2,V3,V4 and V5
• Find coordinator response versions V0 and V1
• TLS
• PLAIN/SASL
• Request / reponse deadlines - socket reads/writes
• Health endpoint
• Prometheus metrics
  1. gauge: proxy_opened_connections {broker}
  2. counter: proxy_requests_total {broker, api_key, api_version}
  3. counter: proxy_connections_total {broker}
  4. counter: proxy_requests_bytes {broker}
  5. counter: proxy_responses_bytes {broker}
• Pluggable proxy authentication
• Deploying Kafka Proxy as a sidecar container
• Performance tests and tuning
• Socket buffer sizing e.g. SO_RCVBUF = 32768, SO_SNDBUF = 131072
• Kafka connect tests
• Different Kafka API versions tests
• Unit and integration tests
• Rolling upgrade test
• Graceful shutdown

Embedded third-party source code

• Cloud SQL Proxy
• Sarama