Skip to content

Taps on kprobes, an alert wording fix, and a README - #1

Merged
necco-c merged 3 commits into
mainfrom
kprobes
Oct 8, 2026
Merged

necco-c merged 3 commits into
mainfrom
kprobes

Conversation

@necco-c

@necco-c necco-c commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

kprobes for every tap, an alert wording fix, and a README

What's in this PR

  • taps: hook tcp_sendmsg/tcp_recvmsg with kprobes (bb4c86a). The socket tap is kprobe/tcp_sendmsg, kprobe/tcp_recvmsg and kretprobe/tcp_recvmsg; every TLS tap's peer_sendmsg/peer_recvmsg are kprobes too. fentry cannot attach on arm64 before 6.4 (AWS Graviton on Amazon Linux 2023 runs 6.1), and a kprobe attaches everywhere. The return probe takes sk from the per-thread stash, the stash slot is cleared at entry, and active_reads is an LRU_HASH, so a return the kretprobe pool skips can never copy from a stale buffer.
  • alerts: name a stopped local port only next to a served API's failure (592a915). Found while capturing the README samples: a third party's unrelated 503 was getting "shop-payments stopped listening" attached to it.
  • docs: README. CLI variant, personas debugging → SRE → DevOps (confirmed by Necco). No license section or badge: the repo is staying private and not open source for now.

Kernel matrix on kprobes: green on 6.1, 6.6, 6.12, 6.18, 7.2 and bpf-next. Attach and capture run by hand on arm64 6.1 (Debian 12) and 6.12 (Debian 13).

✅ Reviewer action items

  • Merge this PR (humans merge).
  • About description set (below).
  • Topic tags applied (below).
  • Flagged claims resolved: 9 flagged in the draft, 0 left. How each was settled is below.
  • Live Slack test, about five minutes together: approve a yeet login link on a test host, connect Slack at yeet.cx/settings, pick a channel, run --test-alert. The README describes what --test-alert does; a real post has not been made yet.
  • Hero GIF: the <img> is commented out. A --discover run on the shop test box is the recording.
  • Optional: a demo/ directory with the shop workload used for every sample, so "Try it without real traffic" can point at it instead of python3 -m http.server.

About description

eBPF API watchdog for Linux: lists the HTTP APIs a box serves and calls, and alerts Slack when one breaks

105 characters, no em-dash, leads with the definition's category.

Topic tags

ebpf
linux
observability
yeet
http
api
alerting
slack
kprobes
uprobes
tls
api-monitoring

How the flagged claims were resolved

claim resolution
Overhead Re-measured on the kprobes build: JS 1.0 s CPU in 5 min (~0.3% of a core), heap 3.6 to 8.9 MiB, ~7 exchanges/s. Kernel side with kernel.bpf_stats_enabled=1: 16.3 ms of probe run time in 120 s (~0.014% of a core; on_sendmsg ~2.1 µs, on_recvmsg_* ~0.8 µs), stated as excluding kprobe firing cost.
TLS runtimes Verified Python urllib, curl, and Node 20 fetch (Debian's package, links system libssl; 8/8 calls read, from a process started after apiwatch). Ruby, PHP, nginx and Deno/Bun claims removed.
Service update Verified on the VM: restart alone keeps the old copy; stop → unit remove → unit add → start runs the new code; a --dev service picks it up on restart. Both written in.
Sign-in then alerts Replaced "works without a restart" with an explicit yeet service restart apiwatch and the "signedIn":true check.
--test-alert Reworded to what the code does (sends through yeet.alert; prints why and exits non-zero on failure, which is verified signed-out). The live post is the open item above.
BTF / kprobe config by distro Replaced distro claims with checks the reader runs (/sys/kernel/btf/vmlinux, grep of the kernel config).
x86_64 attach Reworded: the matrix runs on x86_64 and verifies; attach was run by hand on arm64 only.
License Removed with the section and badge, since the repo is not being open sourced.

Limits stated in the README that are new to the corpus

  • Capture-all cost. Without --ports the socket tap copies every TCP call and the JS discards non-HTTP, which contradicts the corpus's canonical overhead answer; the README says so and points busy hosts at --ports.
  • gRPC errors on HTTP 200 (grpc-status trailers) are not alerts; deferred to grpcsnoop.
  • No-response failures (refused, timeout) produce no alert of their own.
  • kretprobe instance pool: under heavy tcp_recvmsg concurrency some reads are skipped, never miswritten.
  • Restart forgets: a port already down at start is unknown unless named with --ports.

yeet findings worth an issue

  • yeet service unit add -I gh:... clones without running make, so an eBPF tool restart-loops on a missing .bpf.o, and yeet service start blocks while it does. The README's recipe builds first.
  • yeet run . never builds a local directory, only remote sources. Covered in the FAQ.
  • yeet service unit remove on a running service fails silently in a script; the following unit add then says "Unit 'watch' is taken". Stopping first works.

Handoff

repo: apiwatch
variant: cli
mode: one-shot probe (--discover) + long-running monitor (--watch)
readme: ~/code/yeet-scripts-readmes/apiwatch/README.md
personas: [debugging, sre, devops]
category: eBPF API watchdog for Linux
visibility: private, not open source (Necco, 2026-10-08)
defers_to: {httpscope: "API schemas, drift, Go crypto/tls", grpcsnoop: "gRPC errors and messages", container-traffic: "per-container RED dashboard", "UptimeRobot / Pingdom": "external reachability", "Datadog Synthetics": "scripted multi-step checks"}
flagged_claims: 0
uncovered_topics: ["how to read HTTPS plaintext with uprobes on SSL_read/SSL_write", "why loopback hops are counted twice by socket-level capture", "kprobe vs fentry on arm64 before 6.4", "alerting on real traffic vs synthetic health checks", "naming TLS calls from the ClientHello SNI"]

@necco-c
necco-c merged commit f943249 into main Oct 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant