No packet saved in file

[zhangbo1882]

It seems the key is saved but no packet is saved.

[Tue Sep 13 18:42:59][#14# ] (master)$sudo ./bin/ecapture tls --port 8443 -w 1.pcap

tls_2022/09/13 18:43:47 ECAPTURE :: version :linux_x86_64:0.4.3-20220911-b57ba45:5.4.0-59-generic
tls_2022/09/13 18:43:47 ECAPTURE :: pid info :2628333
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL module initialization
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL Module.Run()
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL TC MODEL
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libssl.so.1.1
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL Ifname:eth0, Ifindex:2, Port:8443, Pcapng filepath:/mnt/go/src/github.com/zhangbo1882/ecapture/1.pcap
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL target all process.
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL target all users.
tls_2022/09/13 18:43:47 EBPFProbeOPENSSL module started successfully.
tls_2022/09/13 18:43:47 EBPFProbeGNUTLS module initialization
tls_2022/09/13 18:43:47 EBPFProbeGNUTLS Module.Run()
tls_2022/09/13 18:43:47 EBPFProbeGNUTLS HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
tls_2022/09/13 18:43:47 EBPFProbeGNUTLS target all process.
tls_2022/09/13 18:43:47 EBPFProbeGNUTLS module started successfully.
tls_2022/09/13 18:43:47 EBPFProbeNSPR module initialization
tls_2022/09/13 18:43:47 EBPFProbeNSPR Module.Run()
tls_2022/09/13 18:43:47 EBPFProbeNSPR HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libnspr4.so
tls_2022/09/13 18:43:47 EBPFProbeNSPR target all process.
tls_2022/09/13 18:43:47 EBPFProbeNSPR module started successfully.
tls_2022/09/13 18:43:47 EBPFProbeGoSSL module [disabled].
tls_2022/09/13 18:43:47 ECAPTURE :: start 3 modules
tls_2022/09/13 18:43:50 TLS1_2_VERSION: save CLIENT_RANDOM 2af86376bb2bfada9b15cc55ce017f21b97e00b6ca08d7afc65bb9df3e3ac6b6 to file success, 176 bytes
tls_2022/09/13 18:43:52 TLS1_2_VERSION: save CLIENT_RANDOM f2e3b243844f801aa9bb8cb93f297639ac163a67695914005948ec0781ad24d5 to file success, 176 bytes
^Ctls_2022/09/13 18:43:59 EBPFProbeNSPR close
tls_2022/09/13 18:43:59 EBPFProbeOPENSSL saving pcapng file /mnt/go/src/github.com/zhangbo1882/ecapture/1.pcap
tls_2022/09/13 18:43:59 EBPFProbeOPENSSL save 0 packets into pcapng file.
tls_2022/09/13 18:43:59 EBPFProbeOPENSSL close.
tls_2022/09/13 18:43:59 EBPFProbeOPENSSL close
tls_2022/09/13 18:43:59 EBPFProbeGNUTLS close

[cfc4n]

make sure that there are TCP packets from port 8443 . CLIENT_RANDROM is captured from ebpf uprobe, Can not filter by port argument.

[zhangbo1882]

I am sure there is traffic to 8443.

[blaisewang]

I got the same issue as op. I added the following code for debugging

diff --git a/kern/common.h b/kern/common.h
index e98a079..a181b0b 100644
--- a/kern/common.h
+++ b/kern/common.h
@@ -11,6 +11,12 @@
 #define debug_bpf_printk(fmt, ...)
 #endif

+#define bpf_trace(fmt, ...)                            \
+    do {                                               \
+        char s[] = fmt;                                \
+        bpf_trace_printk(s, sizeof(s), ##__VA_ARGS__); \
+    } while (0)
+
 #define TASK_COMM_LEN 16
 #define MAX_DATA_SIZE_OPENSSL 1024 * 4
 #define MAX_DATA_SIZE_MYSQL 256
diff --git a/kern/openssl_tc.h b/kern/openssl_tc.h
index 9e3f8db..34b33b8 100644
--- a/kern/openssl_tc.h
+++ b/kern/openssl_tc.h
@@ -110,6 +110,8 @@ int capture_packets(struct __sk_buff *skb, bool is_ingress) {
     }
     struct tcphdr *tcp = (struct tcphdr *)(data_start + l4_hdr_off);

+    bpf_trace("got tcphdr");
+
     if (tcp->source != bpf_htons(target_port) &&
         tcp->dest != bpf_htons(target_port)) {
         return TC_ACT_OK;
@@ -138,6 +140,7 @@ int capture_packets(struct __sk_buff *skb, bool is_ingress) {
     size_t pkt_size = TC_PACKET_MIN_SIZE;
     bpf_perf_event_output(skb, &skb_events, flags, &event, pkt_size);

+    bpf_trace("new packet captured on egress/ingress (TC), length:%d\n", data_len);
     //    debug_bpf_printk("new packet captured on egress/ingress (TC),
     //    length:%d\n", data_len);
     return TC_ACT_OK;

When there are no packets saved in the pcapng file, "new packet captured on egress/ingress" will not be printed in /sys/kernel/debug/tracing/trace_pipe. But tons of "got tcphdr" were printed.

@cfc4n any idea?

[blaisewang]

Oh, I got my problem. ecap were captured on, in my case, the eth0 interface. I tested with curl https://127.0.0.1 -k, which was handled by the lo interface.

Did you also use a localhost to test with? Or your 8443 port wasn't on the eth0 interface, for example, say the docker0? @zhangbo1882

[zhangbo1882]

[Tue Sep 20 21:31:15][#6# ] (master)$netstat -atn | grep 8443
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN

[Tue Sep 20 21:33:09][#8# ] (master)$ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.147.1.61 netmask 255.255.252.0 broadcast 10.147.3.255
inet6 fe80::76db:d1ff:fe62:f728 prefixlen 64 scopeid 0x20
ether 74:db:d1:62:f7:28 txqueuelen 1000 (Ethernet)
RX packets 116213491 bytes 89447762492 (89.4 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 35216516 bytes 37161152396 (37.1 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Actually it listened on all interfaces.

I use curl tool in another machine.
[Wed Sep 21 12:33:45][#312# ]$curl -k https://10.147.1.61:8443
Request Headers:
':authority', '10.147.1.61:8443'
':path', '/'
':method', 'GET'
':scheme', 'https'
'user-agent', 'curl/7.79.1'
'accept', '/'
'x-forwarded-for', '10.249.74.45'
'x-forwarded-proto', 'https'
'x-request-id', '8f52b1fa-301c-47ea-b371-325e0aeabe27'
'x-envoy-internal', 'true'

[Tue Sep 20 21:30:57][#5# ] (master)$sudo ./bin/ecapture tls -w 1.pcap
tls_2022/09/20 21:31:07 ECAPTURE :: version :linux_x86_64:0.4.3-20220911-b57ba45:5.4.0-59-generic
tls_2022/09/20 21:31:07 ECAPTURE :: pid info :1258771
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL module initialization
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL Module.Run()
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL TC MODEL
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libssl.so.1.1
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL Ifname:eth0, Ifindex:2, Port:443, Pcapng filepath:/mnt/go/src/github.com/zhangbo1882/ecapture/1.pcap
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL target all process.
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL target all users.
tls_2022/09/20 21:31:07 EBPFProbeOPENSSL module started successfully.
tls_2022/09/20 21:31:07 EBPFProbeGNUTLS module initialization
tls_2022/09/20 21:31:07 EBPFProbeGNUTLS Module.Run()
tls_2022/09/20 21:31:07 EBPFProbeGNUTLS HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
tls_2022/09/20 21:31:07 EBPFProbeGNUTLS target all process.
tls_2022/09/20 21:31:07 EBPFProbeGNUTLS module started successfully.
tls_2022/09/20 21:31:07 EBPFProbeNSPR module initialization
tls_2022/09/20 21:31:07 EBPFProbeNSPR Module.Run()
tls_2022/09/20 21:31:07 EBPFProbeNSPR HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libnspr4.so
tls_2022/09/20 21:31:07 EBPFProbeNSPR target all process.
tls_2022/09/20 21:31:07 EBPFProbeNSPR module started successfully.
tls_2022/09/20 21:31:07 EBPFProbeGoSSL module [disabled].
tls_2022/09/20 21:31:07 ECAPTURE :: start 3 modules

^Ctls_2022/09/20 21:31:14 EBPFProbeOPENSSL saving pcapng file /mnt/go/src/github.com/zhangbo1882/ecapture/1.pcap
tls_2022/09/20 21:31:14 EBPFProbeOPENSSL save 0 packets into pcapng file.
tls_2022/09/20 21:31:14 EBPFProbeOPENSSL close.
tls_2022/09/20 21:31:14 EBPFProbeOPENSSL close
tls_2022/09/20 21:31:14 EBPFProbeGNUTLS close
tls_2022/09/20 21:31:15 EBPFProbeNSPR close

[cfc4n]

will fix it this week.

[cfc4n]

On Wireshark:

Edit -> Preferences -> Protocols -> HTTP -> SSL/TLS Ports
Change it from port 443 to add the custom port number your are using for encrypted traffic
For example: 443, 4443, 8443