eBPF debugging tool
eBPF As a powerful subsystem of Linux kernel debugging, there are many corresponding application level debugging tools, such as bpftool/bpftrace , and provides Lua and Python Debugging interface bcc ； Open source tools for kernel information collection and performance analysis SystemTap The eBPF function of the kernel is also used. However, the author hopes to use the debugging function provided by eBPF on embedded devices. One way is to use the open source SDK (e.g yocto , the author has not tried to build these debugging tools automatically (otherwise, manual cross compilation is required); Another method is to peel off the eBPF components provided by a Linux distribution that supports arm64 architecture (such as raspberry pie running debian system) and integrate them into embedded arm64/Linux devices for debugging; The last method is to write C code with reference to the BPF example provided by the Linux kernel (the code path is samples/bpf) to realize the required debugging functions.
This paper records the process of cross compiling BPF examples for arm64/Linux devices on PC, as well as the problems encountered in the process and simple solutions; The Linux kernel version used by the author is v5.4.123.
Preparation of related dependent Libraries
First, according to the requirements of the document linux-5.4.123/samples/bpf/README.rst, in x86_ Installing on a 64 / Linux host clang and llvm Related tool chains:
sudo apt install clang llvm
The tool chain is not a cross compiler and is only used to generate BTF Commissioning documents; BTF file itself is an ELF file, which contains debugging information related to eBPF. Using BTF can achieve "one compilation, run everywhere" to a certain extent. After that, configure the kernel source code of the embedded device and compile the kernel:
alias lmake='make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-' lmake menuconfig lmake headers_install lmake Image
The installation and compilation of the Linux kernel header file is to generate the corresponding header file in the kernel source code, otherwise the subsequent samples/bpf compilation will make an error. Next, you need to prepare the dependency library for samples/bpf cross compilation. Here you just need to cross compile zlib and elfutils ； The author compiles the two open source libraries in Last article It has been involved in and will not be repeated; It is worth emphasizing that the installation path of these two libraries is / opt/libbpf. In addition, during the compilation of samples/bpf, the kernel compilation script will automatically compile tools/lib/bpf and generate the static library libbpf.a; The debug application in the BPF example is linked to the static library.
Compile all samples in samples/bpf
Before compiling the BPF example, you need to modify the relevant Makefile. The author's modifications are as follows:
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 6d1df7117..f8b1e04e6 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -171,6 +171,7 @@ always += ibumad_kern.o
always += hbm_out_kern.o
always += hbm_edt_kern.o
+KBUILD_HOSTCFLAGS += -I/opt/libbpf/include
KBUILD_HOSTCFLAGS += -I$(objtree)/usr/include
KBUILD_HOSTCFLAGS += -I$(srctree)/tools/lib/bpf/
KBUILD_HOSTCFLAGS += -I$(srctree)/tools/testing/selftests/bpf/
@@ -180,7 +181,7 @@ KBUILD_HOSTCFLAGS += -DHAVE_ATTR_TEST=0
HOSTCFLAGS_bpf_load.o += -I$(objtree)/usr/include -Wno-unused-variable
-KBUILD_HOSTLDLIBS += $(LIBBPF) -lelf
+KBUILD_HOSTLDLIBS += $(LIBBPF) -lelf -L/opt/libbpf/lib -Wl,-rpath-link=/opt/libbpf/lib
HOSTLDLIBS_tracex4 += -lrt
HOSTLDLIBS_trace_output += -lrt
HOSTLDLIBS_map_perf_test += -lrt
diff --git a/tools/build/Makefile.build b/tools/build/Makefile.build
index cd72016c3..7eeadf644 100644
--- a/tools/build/Makefile.build
+++ b/tools/build/Makefile.build
@@ -87,10 +87,6 @@ quiet_cmd_host_ld_multi = HOSTLD $@
cmd_host_ld_multi = $(if $(strip $(obj-y)),\
$(HOSTLD) -r -o $@ $(filter $(obj-y),$^),rm -f $@; $(HOSTAR) rcs $@)
-ifneq ($(filter $(obj),$(hostprogs)),)
- host = host_
-endif
-
# Build rules
$(OUTPUT)%.o: %.c FORCE
$(call rule_mkdir)
diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile
index 9758bfa59..5c2b4ce78 100644
--- a/tools/lib/bpf/Makefile
+++ b/tools/lib/bpf/Makefile
@@ -191,7 +191,8 @@ $(OUTPUT)libbpf.so: $(OUTPUT)libbpf.so.$(LIBBPF_VERSION)
$(OUTPUT)libbpf.so.$(LIBBPF_VERSION): $(BPF_IN_SHARED)
$(QUIET_LINK)$(CC) --shared -Wl,-soname,libbpf.so.$(LIBBPF_MAJOR_VERSION) \
- -Wl,--version-script=$(VERSION_SCRIPT) $^ -lelf -o $@
+ -Wl,--version-script=$(VERSION_SCRIPT) $^ -lelf -o $@ \
+ -L$(prefix)/lib -Wl,-rpath-link=$(prefix)/lib
@ln -sf $(@F) $(OUTPUT)libbpf.so
@ln -sf $(@F) $(OUTPUT)libbpf.so.$(LIBBPF_MAJOR_VERSION)
@@ -199,7 +200,8 @@ $(OUTPUT)libbpf.a: $(BPF_IN_STATIC)
$(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^
$(OUTPUT)test_libbpf: test_libbpf.cpp $(OUTPUT)libbpf.a
- $(QUIET_LINK)$(CXX) $(INCLUDES) $^ -lelf -o $@
+ $(QUIET_LINK)$(CXX) $(INCLUDES) $^ -lelf -o $@ \
+ -L$(prefix)/lib -Wl,-rpath-link=$(prefix)/lib
$(OUTPUT)libbpf.pc:
$(QUIET_GEN)sed -e "s|@PREFIX@|$(prefix)|" \
Some of the above changes have also appeared in the previous article. Finally, a very important step is to compile samples/bpf; Execute in the root directory of Linux kernel source code:
lmake M=samples/bpf prefix=/opt/libbpf \ FEATURE_CHECK_CFLAGS-libelf='-I/opt/libbpf/include' \ FEATURE_CHECK_LDFLAGS-libelf='-L/opt/libbpf/lib -Wl,-rpath-link=/opt/libbpf/lib' \ EXTRA_CFLAGS='-Wall -fPIC -O2 -mcpu=cortex-a53 -I/opt/libbpf/include'
After the command is executed, you can run the BPF sample on the arm64/Linux device. Below, the author records two problems in the debugging process.
Run the samples/bpf sample
The author copied two files to the device, tracex1 and tracex1_kern.o； The former is used to load and debug the latter into the Linux kernel, and the latter is a BTF file generated by the clang compiler. The commissioning results are as follows:
# export LD_LIBRARY_PATH=/opt/libbpf/lib::/opt/libbpf/lib64 # ./tracex1 invalid relo for insn[4].code 0x85 bpf_load_program() err=22 last insn is not an exit or jmp processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 last insn is not an exit or jmp processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
As a result, BTF file loading failed. Use llvm objdump to disassemble tracex1_kern.o can see that there is an eBPF assembly instruction of call -1:
$ llvm-objdump -d tracex1_kern.o
tracex1_kern.o: file format ELF64-BPF
Disassembly of section kprobe/__netif_receive_skb_core:
0000000000000000 bpf_prog1:
0: 79 13 00 00 00 00 00 00 r3 = *(u64 *)(r1 + 0)
1: bf a1 00 00 00 00 00 00 r1 = r10
2: 07 01 00 00 e8 ff ff ff r1 += -24
3: b7 02 00 00 08 00 00 00 r2 = 8
4: 85 10 00 00 ff ff ff ff call -1
5: b7 06 00 00 00 00 00 00 r6 = 0
6: 7b 6a f0 ff 00 00 00 00 *(u64 *)(r10 - 16) = r6
7: 79 a3 e8 ff 00 00 00 00 r3 = *(u64 *)(r10 - 24)
8: 07 03 00 00 10 00 00 00 r3 += 16
This partly means that references to missing symbols are usually undefined functions. View tracex1_kern.c source code, BPF_ The prog1 function called bpf_probe_read_kernel:
/* linux-5.4.123/samples/bpf/tracex1_kern.c */
SEC("kprobe/__netif_receive_skb_core")
int bpf_prog1(struct pt_regs *ctx)
{
/* attaches to kprobe __netif_receive_skb_core,
* looks for packets on loobpack device and prints them
*/
char devname[IFNAMSIZ];
struct net_device *dev;
struct sk_buff *skb;
int len;
/* non-portable! works for the given kernel only */
bpf_probe_read_kernel(&skb, sizeof(skb), (void *)PT_REGS_PARM1(ctx));
dev = _(skb->dev);
len = _(skb->len);
When searching the entire Linux kernel source code, the function is undefined and can only be searched once:
$ git grep -n bpf_probe_read_kernel samples/bpf/tracex1_kern.c:32: bpf_probe_read_kernel(&skb, sizeof(skb), (void *)PT_REGS_PARM1(ctx));
This indicates that some examples under samples/bpf are not available. In fact, the eBPF kernel interface provided by the Linux kernel is constantly changing, and this problem is normal; From the side, it reflects that active kernel development activities have been in progress.
The author tries to debug tracex2 and tracex2_kern.o, the result is abnormal again:
# ./tracex2 failed to create kprobe 'sys_write' error 'No such file or directory'
The reason for this problem is that on arm64/Linux devices, the function name of the system call write in the kernel is not sys_write for__ arm64_sys_write； You can confirm by reading / proc/kallsyms:
# cat /proc/kallsyms | grep -e sys_write ffffffc0102a1d28 T __arm64_sys_writev ffffffc0102a2090 T __arm64_compat_sys_writev ffffffc0102a4630 T ksys_write ffffffc0102a46f0 T __arm64_sys_write ffffffc010346738 t proc_sys_write
This naming difference stems from the compatibility mechanism of Linux kernel for different CPU architectures. Repair tracex2_kern.c code:
diff --git a/samples/bpf/tracex2_kern.c b/samples/bpf/tracex2_kern.c
index 5e11c20ce..0c5330fb2 100644
--- a/samples/bpf/tracex2_kern.c
+++ b/samples/bpf/tracex2_kern.c
@@ -76,7 +76,7 @@ struct bpf_map_def SEC("maps") my_hist_map = {
.max_entries = 1024,
};
-SEC("kprobe/sys_write")
+SEC("kprobe/__arm64_sys_write")
int bpf_prog3(struct pt_regs *ctx)
{
long write_size = PT_REGS_PARM3(ctx);
Then compile again to generate tracex2_kern.o, running on the device can get the correct example running results:
# ./tracex2
location 0xa94153f352800020 count 1
location 0xa94153f352800020 count 2
location 0xa94153f352800020 count 3
location 0xa94153f352800020 count 4
pid 1 cmd procd uid 0
syscall write() stats
byte_size : count distribution
1 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 0 | |
4096 -> 8191 : 0 | |
8192 -> 16383 : 0 | |
16384 -> 32767 : 0 | |
32768 -> 65535 : 0 | |
65536 -> 131071 : 0 | |
131072 -> 262143 : 0 | |
262144 -> 524287 : 0 | |
524288 -> 1048575 : 0 | |
1048576 -> 2097151 : 0 | |
2097152 -> 4194303 : 0 | |
4194304 -> 8388607 : 0 | |
8388608 -> 16777215 : 0 | |
16777216 -> 33554431 : 0 | |
33554432 -> 67108863 : 0 | |
67108864 -> 134217727 : 0 | |
134217728 -> 268435455 : 0 | |
268435456 -> 536870911 : 0 | |
536870912 -> 1073741823 : 0 | |
1073741824 -> 2147483647 : 0 | |
2147483648 -> 4294967295 : 0 | |
4294967296 -> 8589934591 : 0 | |
8589934592 -> 17179869183 : 0 | |
17179869184 -> 34359738367 : 0 | |
34359738368 -> 68719476735 : 0 | |
68719476736 -> 137438953471 : 0 | |
137438953472 -> 274877906943 : 0 | |
274877906944 -> 549755813887 : 0 | |
549755813888 -> 1099511627775 : 0 | |
1099511627776 -> 2199023255551 : 0 | |
2199023255552 -> 4398046511103 : 0 | |
4398046511104 -> 8796093022207 : 0 | |
8796093022208 -> 17592186044415 : 0 | |
17592186044416 -> 35184372088831 : 0 | |
35184372088832 -> 70368744177663 : 0 | |
70368744177664 -> 140737488355327 : 0 | |
140737488355328 -> 281474976710655 : 0 | |
281474976710656 -> 562949953421311 : 0 | |
562949953421312 -> 1125899906842623 : 0 | |
1125899906842624 -> 2251799813685247 : 0 | |
2251799813685248 -> 4503599627370495 : 0 | |
4503599627370496 -> 9007199254740991 : 0 | |
9007199254740992 -> 18014398509481983 : 0 | |
18014398509481984 -> 36028797018963967 : 0 | |
36028797018963968 -> 72057594037927935 : 0 | |
72057594037927936 -> 144115188075855871 : 0 | |
144115188075855872 -> 288230376151711743 : 0 | |
288230376151711744 -> 576460752303423487 : 0 | |
576460752303423488 -> 1152921504606846975 : 0 | |
1152921504606846976 -> 2305843009213693951 : 0 | |
2305843009213693952 -> 4611686018427387903 : 0 | |
-4611686018427387904 -> 9223372036854775807 : 0 | |
0 -> 0 : 1 |************************************* |
......
Paying attention to these two problems can help deepen the understanding of BPF functions and facilitate the learning of BPF kernel debugging functions on embedded devices.