-
Notifications
You must be signed in to change notification settings - Fork 57
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add a chapter about logging custom structs with PerfEventArray
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
- Loading branch information
1 parent
54f1609
commit c69999c
Showing
22 changed files
with
656 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,161 @@ | ||
# Getting data to user-space | ||
|
||
In previous chapters we were logging packets with aya-log. However, what if we | ||
need to send additional data about the packet or any other type of information | ||
for the user-space program to utilize? In this chapter, we will explore how to | ||
leverage perf buffers and define data structures in the *-common* crate to | ||
transfer data from eBPF program to user-space applications. By doing so, | ||
user-space programs can access and utilize the transferred data effectively. | ||
|
||
!!! example "Source code" | ||
|
||
Full code for the example in this chapter is available | ||
[here](https://github.com/aya-rs/book/tree/main/examples/xdp-perfbuf-custom-data) | ||
|
||
## Sharing data | ||
|
||
In this chapter, we'll be sending data from the kernel-space to the user-space | ||
by writing it into a struct and outputting it to user-space. We can achieve this | ||
by using an eBPF map. There are different types of maps available, but in this | ||
case, we'll use `PerfEventArray`. | ||
|
||
`PerfEventArray` is a collection of per-CPU circular buffers that enable the | ||
kernel to emit events (defined as custom structs) to user-space. Each CPU has | ||
its own buffer, and the eBPF program emits an event to the buffer of the CPU | ||
it's currently running on. The events are out of order, meaning that they arrive | ||
in the user-space in a different order than they were created and sent from the | ||
eBPF program. | ||
|
||
To gather events from all CPUs, a user-space program needs to spawn a thread for | ||
each CPU to poll for the events and then iterate over them. | ||
|
||
The data structure we'll be using needs to hold an IPv4 address and a port. | ||
|
||
```rust linenums="1" title="xdp-perfbuf-custom-data-common/src/lib.rs" | ||
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-common/src/lib.rs" | ||
``` | ||
|
||
1. We implement the `aya::Pod` trait for our struct since it is Plain Old Data | ||
as can be safely converted to a byte-slice and back. | ||
|
||
!!! tip "Alignment, padding and verifier errors" | ||
|
||
At program load time, the eBPF verifier checks that all the memory used is | ||
properly initialized. This can be a problem if - to ensure alignment - the | ||
compiler inserts padding bytes between fields in your types. | ||
|
||
**Example:** | ||
|
||
```rust | ||
#[repr(C)] | ||
struct SourceInfo { | ||
source_port: u16, | ||
source_ip: u32, | ||
} | ||
|
||
let source_port = ...; | ||
let source_ip = ...; | ||
let si = SourceInfo { source_port, source_ip }; | ||
``` | ||
|
||
In the example above, the compiler will insert two extra bytes between the | ||
struct fields `source_port` and `source_ip` to make sure that `source_ip` is | ||
correctly aligned to a 4 bytes address (assuming `mem::align_of::<u32>() == | ||
4`). Since padding bytes are typically not initialized by the compiler, | ||
this will result in the infamous `invalid indirect read from stack` verifier | ||
error. | ||
|
||
To avoid the error, you can either manually ensure that all the fields in | ||
your types are correctly aligned (e.g. by explicitly adding padding or by | ||
making field types larger to enforce alignment) or use `#[repr(packed)]`. | ||
Since the latter comes with its own foot-guns and can perform less | ||
efficiently, explicitly adding padding or tweaking alignment is recommended. | ||
|
||
**Solution ensuring alignment using larger types:** | ||
|
||
```rust | ||
#[repr(C)] | ||
struct SourceInfo { | ||
source_port: u32, | ||
source_ip: u32, | ||
} | ||
|
||
let source_port = ...; | ||
let source_ip = ...; | ||
let si = SourceInfo { source_port, source_ip }; | ||
``` | ||
|
||
**Solution with explicit padding:** | ||
|
||
```rust | ||
#[repr(C)] | ||
struct SourceInfo { | ||
source_port: u16, | ||
padding: u16, | ||
source_ip: u32, | ||
} | ||
|
||
let source_port = ...; | ||
let source_ip = ...; | ||
let si = SourceInfo { source_port, padding: 0, source_ip }; | ||
``` | ||
|
||
## Getting packet data from the context and into the map | ||
|
||
Our eBPF program code is going to be similar to the one in the previous chapters. | ||
It's going to get the information about source IP address and port from packet | ||
headers. | ||
|
||
The difference is that once we get the data from headers, instead of logging | ||
them, we create a `PacketLog` struct and output it to our `PerfEventArray` | ||
|
||
The resulting code looks like this: | ||
|
||
```rust linenums="1" title="xdp-perfbuf-custom-data-ebpf/src/main.rs" | ||
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/src/main.rs" | ||
``` | ||
|
||
1. Create our map. | ||
2. Output the event to the map. | ||
|
||
Don't forget to rebuild your eBPF program! | ||
|
||
## Reading data | ||
|
||
In order to read from the `AsyncPerfEventArray`, we have to call | ||
`AsyncPerfEventArray::open()` for each online CPU, then we have to poll the file | ||
descriptor for events. While this is do-able using `PerfEventArray` and `mio` or | ||
`epoll`, the code is much less easy to follow. Instead, we'll use `tokio`, which | ||
was added to our template for us. | ||
|
||
We'll need to add a dependency on `bytes` to `xdp-log/Cargo.toml` since | ||
this will make it easier to deal with the chunks of bytes yielded by the | ||
`AsyncPerfEventArray`. | ||
|
||
Here's the code: | ||
|
||
```rust linenums="1" title="xdp-perfbuf-custom-data/src/main.rs" | ||
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data/src/main.rs" | ||
``` | ||
|
||
1. Define our map. | ||
2. Call `open()` for each online CPU. | ||
3. Spawn a `tokio::task`. | ||
4. Create buffers. | ||
5. Read events in to buffers. | ||
6. Use `read_unaligned` to read the event data into a `PacketLog`. | ||
7. Log the packet data to the console. | ||
|
||
## Running the program | ||
|
||
As before, the interface can be overwritten by providing the interface name as a | ||
parameter, for example, `RUST_LOG=info cargo xtask run -- --iface wlp2s0`. | ||
|
||
```console | ||
$ RUST_LOG=info cargo xtask run | ||
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 60.235.240.157, SRC_PORT: 443 | ||
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 98.21.76.76, SRC_PORT: 443 | ||
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443 | ||
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443 | ||
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.10.251.142, SRC_PORT: 443 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
[alias] | ||
xtask = "run --package xtask --" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
### https://raw.github.com/github/gitignore/master/Rust.gitignore | ||
|
||
# Generated by Cargo | ||
# will have compiled files and executables | ||
debug/ | ||
target/ | ||
|
||
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries | ||
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html | ||
Cargo.lock | ||
|
||
# These are backup files generated by rustfmt | ||
**/*.rs.bk |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"rust-analyzer.linkedProjects": ["Cargo.toml", "xdp-perfbuf-custom-data-ebpf/Cargo.toml"] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"rust-analyzer.linkedProjects": ["Cargo.toml", "xdp-perfbuf-custom-data-ebpf/Cargo.toml"] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
[workspace] | ||
members = ["xdp-perfbuf-custom-data", "xdp-perfbuf-custom-data-common", "xtask"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
# xdp-perfbuf-custom-data | ||
|
||
## Prerequisites | ||
|
||
1. Install a rust stable toolchain: `rustup install stable` | ||
1. Install a rust nightly toolchain: `rustup install nightly` | ||
1. Install bpf-linker: `cargo install bpf-linker` | ||
|
||
## Build eBPF | ||
|
||
```bash | ||
cargo xtask build-ebpf | ||
``` | ||
|
||
To perform a release build you can use the `--release` flag. | ||
You may also change the target architecture with the `--target` flag | ||
|
||
## Build Userspace | ||
|
||
```bash | ||
cargo build | ||
``` | ||
|
||
## Run | ||
|
||
```bash | ||
RUST_LOG=info cargo xtask run | ||
``` |
14 changes: 14 additions & 0 deletions
14
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-common/Cargo.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
[package] | ||
name = "xdp-perfbuf-custom-data-common" | ||
version = "0.1.0" | ||
edition = "2021" | ||
|
||
[features] | ||
default = [] | ||
user = [ "aya" ] | ||
|
||
[dependencies] | ||
aya = { version = ">=0.11", optional=true } | ||
|
||
[lib] | ||
path = "src/lib.rs" |
11 changes: 11 additions & 0 deletions
11
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-common/src/lib.rs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
#![no_std] | ||
|
||
#[repr(C)] | ||
#[derive(Clone, Copy)] | ||
pub struct PacketLog { | ||
pub ipv4_address: u32, | ||
pub port: u32, | ||
} | ||
|
||
#[cfg(feature = "user")] | ||
unsafe impl aya::Pod for PacketLog {} // (1) |
6 changes: 6 additions & 0 deletions
6
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/.cargo/config.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
[build] | ||
target-dir = "../target" | ||
target = "bpfel-unknown-none" | ||
|
||
[unstable] | ||
build-std = ["core"] |
4 changes: 4 additions & 0 deletions
4
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/.vim/coc-settings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"rust-analyzer.cargo.target": "bpfel-unknown-none", | ||
"rust-analyzer.checkOnSave.allTargets": false | ||
} |
4 changes: 4 additions & 0 deletions
4
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/.vscode/settings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"rust-analyzer.cargo.target": "bpfel-unknown-none", | ||
"rust-analyzer.checkOnSave.allTargets": false | ||
} |
33 changes: 33 additions & 0 deletions
33
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/Cargo.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
[package] | ||
name = "xdp-perfbuf-custom-data-ebpf" | ||
version = "0.1.0" | ||
edition = "2021" | ||
|
||
[dependencies] | ||
aya-bpf = { git = "https://github.com/aya-rs/aya", branch = "main" } | ||
aya-log-ebpf = { git = "https://github.com/aya-rs/aya", branch = "main" } | ||
xdp-perfbuf-custom-data-common = { path = "../xdp-perfbuf-custom-data-common" } | ||
network-types = "0.0.4" | ||
|
||
[[bin]] | ||
name = "xdp-perfbuf-custom-data" | ||
path = "src/main.rs" | ||
|
||
[profile.dev] | ||
opt-level = 3 | ||
debug = false | ||
debug-assertions = false | ||
overflow-checks = false | ||
lto = true | ||
panic = "abort" | ||
incremental = false | ||
codegen-units = 1 | ||
rpath = false | ||
|
||
[profile.release] | ||
lto = true | ||
panic = "abort" | ||
codegen-units = 1 | ||
|
||
[workspace] | ||
members = [] |
5 changes: 5 additions & 0 deletions
5
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/rust-toolchain.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
[toolchain] | ||
channel="nightly-2023-01-10" | ||
# The source code of rustc, provided by the rust-src component, is needed for | ||
# building eBPF programs. | ||
components = [ "rustc", "rust-std", "cargo", "rust-docs", "rustfmt", "clippy", "rust-src" ] |
84 changes: 84 additions & 0 deletions
84
examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/src/main.rs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
#![no_std] | ||
#![no_main] | ||
|
||
use core::mem; | ||
|
||
use aya_bpf::{ | ||
bindings::xdp_action, | ||
macros::{map, xdp}, | ||
maps::PerfEventArray, | ||
programs::XdpContext, | ||
}; | ||
use network_types::{ | ||
eth::{EthHdr, EtherType}, | ||
ip::{IpProto, Ipv4Hdr}, | ||
tcp::TcpHdr, | ||
udp::UdpHdr, | ||
}; | ||
|
||
use xdp_perfbuf_custom_data_common::PacketLog; | ||
|
||
#[map(name = "EVENTS")] // (1) | ||
static mut EVENTS: PerfEventArray<PacketLog> = | ||
PerfEventArray::<PacketLog>::with_max_entries(1024, 0); | ||
|
||
#[xdp(name = "xdp_perfbuf_custom_data")] | ||
pub fn xdp_perfbuf_custom_data(ctx: XdpContext) -> u32 { | ||
match try_xdp_perfbuf_custom_data(ctx) { | ||
Ok(ret) => ret, | ||
Err(_) => xdp_action::XDP_ABORTED, | ||
} | ||
} | ||
|
||
#[inline(always)] | ||
fn ptr_at<T>(ctx: &XdpContext, offset: usize) -> Result<*const T, ()> { | ||
let start = ctx.data(); | ||
let end = ctx.data_end(); | ||
let len = mem::size_of::<T>(); | ||
|
||
if start + offset + len > end { | ||
return Err(()); | ||
} | ||
|
||
Ok((start + offset) as *const T) | ||
} | ||
|
||
fn try_xdp_perfbuf_custom_data(ctx: XdpContext) -> Result<u32, ()> { | ||
let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?; | ||
match unsafe { (*ethhdr).ether_type } { | ||
EtherType::Ipv4 => {} | ||
_ => return Ok(xdp_action::XDP_PASS), | ||
} | ||
|
||
let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)?; | ||
let source_addr = unsafe { (*ipv4hdr).src_addr }; | ||
|
||
let source_port = match unsafe { (*ipv4hdr).proto } { | ||
IpProto::Tcp => { | ||
let tcphdr: *const TcpHdr = | ||
ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN)?; | ||
u16::from_be(unsafe { (*tcphdr).source }) | ||
} | ||
IpProto::Udp => { | ||
let udphdr: *const UdpHdr = | ||
ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN)?; | ||
u16::from_be(unsafe { (*udphdr).source }) | ||
} | ||
_ => return Err(()), | ||
}; | ||
|
||
let log_entry = PacketLog { | ||
ipv4_address: source_addr, | ||
port: source_port as u32, | ||
}; | ||
unsafe { | ||
EVENTS.output(&ctx, &log_entry, 0); // (2) | ||
} | ||
|
||
Ok(xdp_action::XDP_PASS) | ||
} | ||
|
||
#[panic_handler] | ||
fn panic(_info: &core::panic::PanicInfo) -> ! { | ||
unsafe { core::hint::unreachable_unchecked() } | ||
} |
Oops, something went wrong.