Jetson basic notes 2 -- source code compilation and KO generation

Posted by Singularity on Sun, 23 Jan 2022 15:28:14 +0100

Kernel source code compilation

For the Jetson board made by yourself, you generally need to modify the device tree and driver, compile, and then replace the kernel image and device tree. Refer to the following Makefile file:

  • make env_ Dependent, installation environment dependent
  • make download, download the BSP source code (Sources include kernel/u-boot, etc.), and cross compile the tool chain
  • make decompress, decompress the kernel source code and cross compile tool chain
  • make prepare_build_kernel, which prepares for compilation, mainly puts the top-level Makefile and related files into a build folder
  • make build, compile the kernel source code, including make menuconfig. The compiled kernel image is build/arch/arm64/boot/Image, and the device tree blob is build / arch / arm64 / boot / DTS / tegra186-quill-p3310-1000-c03-00-base dtb
  • make scp. You can transfer the Image and dtb to the home directory of the remote TX2
BOARD = jetson-tx2-devkit

PWD := $(shell pwd)

# TX2 Image and dtb
TX2_REMOTE= tx2@192.168.6.157
TX2_IMAGE = ${PWD}/build/arch/arm64/boot/Image
TX2_DTB   = ${PWD}/build/arch/arm64/boot/dts/tegra186-quill-p3310-1000-c03-00-base.dtb

L4T_RELEASE_PACKAGE = jetson_linux_r32.6.1_aarch64.tbz2
SAMPLE_FS_PACKAGE = tegra_linux_sample-root-filesystem_r32.6.1_aarch64.tbz2

CROSS_COMPILE = ${PWD}/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
LOCALVERSION = -tegra

TEGRA_KERNEL_OUT = ${PWD}/build
RELEASE_DIR = ${PWD}/Linux_for_Tegra/kernel


.PHONY: download
download:
	# L4T Driver Package (BSP), ~355MB
	wget https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/t186/jetson_linux_r32.6.1_aarch64.tbz2
	
	# Sample Root Filesystem, ~1430MB
	wget https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/t186/tegra_linux_sample-root-filesystem_r32.6.1_aarch64.tbz2
	
	# Jetson Linux Developer Guide (downloadable version)
	wget https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/nvidia_jetson_linux_driver_package.tar
	
	# L4T Driver Package (BSP) Sources, ~174MB
	wget https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/sources/t186/public_sources.tbz2
	
	# toolchain
	wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

.PHONY: env_depend
env_depend:
	sudo apt update
	sudo apt install -y qemu-user-static
	sudo apt install -y build-essential bc

.PHONY: decompress
decompress:
	@# ===== doc =====
	mkdir -p l4t_docs & tar xvf nvidia_jetson_linux_driver_package.tar -C l4t_docs

	@# ===== bsp =====
	@echo "\n-->\ndecompress l4t release package\n-->\n"
	@echo ${L4T_RELEASE_PACKAGE}
	tar xf ${L4T_RELEASE_PACKAGE}

	@# ===== rootfs =====
	@# must use sudo, or apply_binaries.sh will break and tell you do this 
	@echo "\n--->\ndecompress sample fs package\n--->\n"
	cd Linux_for_Tegra/rootfs/; \
	sudo tar xpf ../../${SAMPLE_FS_PACKAGE}; \
	cd .. ..

	@# ===== sources =====
	@echo "\n--->\nsources\n--->\n"
	tar -xjf public_sources.tbz2
	cd Linux_for_Tegra/source/public; \
	tar xjf kernel_src.tbz2; \
	cd .. .. ..

	@# ===== toolchain =====
	@echo "\n--->\ntoolchain\n--->\n"
	mkdir -p l4t-gcc; \
	cd l4t-gcc; \
	tar xf ../gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz; \
	cd ..


.PHONY: apply_binaries
apply_binaries:
	cd Linux_for_Tegra; \
	sudo ./apply_binaries.sh; \
	cd ..
	@# the last line must be Success!


.PHONY: flashing
flashing:
	cd Linux_for_Tegra; \
	sudo ./flash.sh ${BOARD} mmcblk0p1


.PHONY: prepare_build_kernel
prepare_build_kernel:
	export CROSS_COMPILE=${CROSS_COMPILE}; \
	export LOCALVERSION=${LOCALVERSION}; \
	mkdir -p ${TEGRA_KERNEL_OUT}; \
	cd ${PWD}/Linux_for_Tegra/source/public/kernel/kernel-4.9; \
	make ARCH=arm64 O=${TEGRA_KERNEL_OUT} tegra_defconfig

.PHONY: build
build:
	export CROSS_COMPILE=${CROSS_COMPILE}; \
	export LOCALVERSION=${LOCALVERSION}; \
	export TEGRA_KERNEL_OUT=${TEGRA_KERNEL_OUT}; \
	cd ${TEGRA_KERNEL_OUT}; \
	make menuconfig; \
	make ARCH=arm64 O=${TEGRA_KERNEL_OUT} tegra_defconfig; \
	make ARCH=arm64 O=${TEGRA_KERNEL_OUT} -j $$(nproc); \
	cp ${TEGRA_KERNEL_OUT}/arch/arm64/boot/Image ${RELEASE_DIR}/; \
	cp -r ${TEGRA_KERNEL_OUT}/arch/arm64/boot/dts/* ${RELEASE_DIR}/dtb/


.PHONY: scp
scp:
	sudo scp ${TX2_IMAGE} ${TX2_DTB} ${TX2_REMOTE}:~


.PHONY: ssh
ssh:
	ssh ${TX2_REMOTE}

Kernel and device tree updates

There are many methods:

  • Plug in USB and flash SH, - K update kernel, - d update device tree file
  • Change extlinux Conf, NFS mount?
  • Change extlinux Conf, direct replacement, here in this way

Location reference of DTB file:

You need to set it first

Although by default, L4T is no longer in / boot / extlinux / extlinux If the FDT entry is set in conf, the user can still set this option manually to make the change take effect

# TX2
$ sudo vi /boot/extlinux/extlinux.conf
# add to
FDT /boot/dtb/kernel_tegra186-quill-p3310-1000-c03-00-base.dtb

Use the following

# Update kernel and device tree
# PC
sudo scp build/arch/arm64/boot/Image build/arch/arm64/boot/dts/tegra186-quill-p3310-1000-c03-00-base.dtb tx2@192.168.6.157:~

# TX2, 1.sh
#!/bin/bash
sudo mv ~/Image /boot
sudo mv ~/tegra186-quill-p3310-1000-c03-00-base.dtb /boot/dtb/kernel_tegra186-quill-p3310-1000-c03-00-base.dtb
# sync
sudo reboot

# Note that the dtb name is preceded by a kernel_

Check for updates

# The time should be 12 hours faster than the Ubuntu system time

# kernel
$ uname -a
Linux tx2-pc 4.9.253 #1 SMP PREEMPT Wed Jan 19 22:13:36 PST 2022 aarch64 aarch64 aarch64 GNU/Linux

# Device tree
$ dmesg | grep DTB
[    0.164259] DTB Build time: Jan 19 2022 22:03:52
[    0.433289] DTB Build time: Jan 19 2022 22:03:52

Some components of distribution

As mentioned in previous articles, when solving the problem of installing old distribution drivers on new computers (for example, Ubuntu 16/18 and graphics display / WiFi 6 / Bluetooth on computers released in 2021 / 20222 will certainly be abnormal), it can be supported by updating the kernel. For example, when updating to 5.11 kernel, you need to download:

# https://blog.csdn.net/weifengdq/article/details/118915007
$ tree
.
├── linux-firmware_1.197.2_all.deb
├── linux-headers-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb
├── linux-headers-5.11.0-051100_5.11.0-051100.202102142330_all.deb
├── linux-image-unsigned-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb
└── linux-modules-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb

Of which:

  • Image is the kernel image. After all, the compiled kernel image is running at the end, not the source code
  • Modules are modules that can be dynamically loaded into the kernel ko components, which are often compiled through the source code
  • Firmware is a firmware binary blob that contains some or all functions of some hardware devices. It is usually proprietary, because some hardware manufacturers will not release the source code (such as NVIDIA graphics card, Intel's Wi Fi chipset, trying to avoid the poison of GPL, subway elderly watch. jpg)
  • headers: a software package that provides Linux kernel header files and serves as interfaces between internal kernel components and between user space and the kernel

If you are building a complete kernel, obviously, you need a complete source file, not just a header file.

However, if you are compiling device drivers or other loadable modules linked to the kernel, you only need the header file, so you can save space by not installing the complete source code.

When we write kernel modules, we often see

#include<linux/module.h>
#include<linux/init.h>
#include<linux/kernel.h>

Where are these files? The PC files are actually in the / lib/modules/$(uname -r)/build folder, such as Linux / module H should correspond to / lib / modules / 5.13.0-27-generic / build / include / Linux / module h. There must be Linux in the source code package of TX2_ for_ Tegra/source/public/kernel/kernel-4.9/include/linux/module.h

You can even open make menuconfig directly in the folder, such as the X86 platform of the computer

# ubuntu20, now is 5.13.0-27-generic
cd /lib/modules/$(uname -r)/build
sudo make menuconfig
# you will find Kconfig Makefile here

X86 First KO

Let's write a hello module for x86 platform:

mkdir hello
cd hello
vi hello.c

hello.c source code and comments

//linux headers
#include <linux/module.h>
#include <linux/init.h>
#include <linux/kernel.h>

//Execute on load
static int hello_init(void)
{
    //Print messages to the kernel log buffer, usually viewed with dmesg
    //KERN_ALERT is the log level You can also use pr_alert()
    printk(KERN_ALERT "Hello Module Test\n");
    pr_alert("Hello Module Test2\n");
    return 0;
}

//Execute on uninstall
static void hello_exit(void)
{
    printk(KERN_INFO "Goodbye Hello\n");
}

//Pass to API
//module_init, drive initialization entry point
//module_exit, the entry point that drives the exit
module_init(hello_init);
module_exit(hello_exit);

//The author declares that with "Name < email >" or "Name", multiple authors can have multiple modules_ Author () line
MODULE_AUTHOR("yiming");
//LICENSE states that it has legal effect and GPL is infectious
MODULE_LICENSE("GPL");
//Description, viewed through modinfo
MODULE_DESCRIPTION("First Driver");

Then vi Makefile

obj-m:=hello.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(shell pwd) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(shell pwd) clean

Of which:

  • -C. change directory, that is, switch to / lib/modules/$(shell uname -r)/build. Here is the kernel build directory of the current Linux version. Here is the top-level Makefile of the kernel
  • M =, a variable of the top-level Makefile refers to the following source code. Let the Makefile move back to the specified directory, pwd, before trying to generate the module target, indicating hello Ko generated in current directory
# From / lib/modules/$(shell uname -r)/build/Makefile

# Use make M=dir or set the environment variable KBUILD_EXTMOD to specify the
# directory of external module to build. Setting M= takes precedence.
ifeq ("$(origin M)", "command line")
  KBUILD_EXTMOD := $(M)
endif

Now start walking

# hello directory
$ tree
.
├── hello.c
└── Makefile

$ make

$ tree
.
├── hello.c
├── hello.ko
├── hello.mod
├── hello.mod.c
├── hello.mod.o
├── hello.o
├── Makefile
├── modules.order
└── Module.symvers

# Check the file type, elf 64bit LSB floating program, x86-64 platform
$ file hello.ko
hello.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=ea5c2968e3fe351b700b5e6f8978904e58254b8f, with debug_info, not stripped

# Load hello ko
$ sudo insmod hello.ko 

# View the last two lines
$ dmesg | tail -2
[23749.761288] Hello Module Test
[23749.761293] Hello Module Test2

# Uninstall hello Ko, no suffix is allowed ko
$ sudo rmmod hello
$ dmesg | tail -1
[23902.056184] Goodbye Hello

# If you let dmesg output directly to the console
# sudo dmesg -n 8 or sudo dmesg -n debug
# Restore default
# sudo dmesg -n 4 or 
# Keep dmesg displayed without exiting
$ dmesg -wH &

The last command can also view the level. The Red alert:

Jetson First KO

Refer to the two sections of kernel customization - > preparing to Building External Kernel Modules and Building External Kernel Modules in the document to compile an external Jetson kernel module:

  • Preprocessing: prepare the header, top-level makefile and kernel in the module source file_ source_ Tree, etc. just prepare it once
  • Set variable CROSS_COMPILE to specify the cross compiler
  • Set - C < top_ Makefile_ Dir > to specify the directory where the top-level Makefile is located, which is usually in the kernel directory, but here I directly threw it into the top folder
  • Set M = specify the location of the module

Create a new Jetson_ Hello folder (/ home/z/jetson/test/jetson_hello), or the above hello C copy it in and then make file

obj-m:=hello.o

kernel_dir = /home/z/jetson/Linux_for_Tegra/source/public/kernel/kernel-4.9
cross_compile_dir = /home/z/jetson/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-

makefile_dir := $(shell pwd)
top_makefile_dir := $(shell pwd)/top
build_dir := $(shell pwd)/build

all:
	@if [ ! -d ${top_makefile_dir} ]; then \
		export CROSS_COMPILE=${cross_compile_dir}; \
		export LOCALVERSION=-tegra; \
		mkdir -p ${top_makefile_dir}; \
		cd ${kernel_dir}; \
		make ARCH=arm64 O=${top_makefile_dir} tegra_defconfig; \
		cd ${top_makefile_dir}; \
		make ARCH=arm64 O=${top_makefile_dir} -j $$(nproc) modules_prepare; \
		echo "\n=================================\n"; \
	fi; \
	export CROSS_COMPILE=${cross_compile_dir}; \
	cd ${makefile_dir};\
	mkdir -p ${build_dir}; \
	make ARCH=arm64 -C ${top_makefile_dir} M=${makefile_dir} -j $$(nproc); \
	mv *.mod.c *.o *.order *.symvers ${build_dir}
	
clean:
	@# rm -rf *.mod.* *.o *.order *.symvers *.ko
	rm -rf build *.ko top

Of which:

  • obj-m, where m refers to module. Generally, it is selected and translated into the kernel (y) in make menuconfig with Kconfig file, or compiled by KO(m), or not compiled (n). Here - M is directly specified
  • When making, first judge whether the top folder exists. If it does not exist, do the above preprocessing step
  • Set the variable cross every time you compile_ Compile to specify the cross compiler
  • -j $(nproc), all CPU s enter the field, but there is only one here c file, not available

Test one

$ make
make[1]: Entering directory '/home/z/jetson/test/jetson_hello/top'

  WARNING: Symbol version dump ./Module.symvers
           is missing; modules will have no dependencies and modversions.

  LD      /home/z/jetson/test/jetson_hello/built-in.o
  CC [M]  /home/z/jetson/test/jetson_hello/hello.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/z/jetson/test/jetson_hello/hello.mod.o
  LD [M]  /home/z/jetson/test/jetson_hello/hello.ko
make[1]: Leaving directory '/home/z/jetson/test/jetson_hello/top'

$ tree -L 2
.
├── build
│   ├── built-in.o
│   ├── hello.mod.c
│   ├── hello.mod.o
│   ├── hello.o
│   ├── modules.order
│   └── Module.symvers
├── hello.c
├── hello.ko
├── Makefile
└── top
    ├── arch
    ├── include
    ├── kernel
    ├── Makefile
    ├── scripts
    └── source -> /home/z/jetson/Linux_for_Tegra/source/public/kernel/kernel-4.9

# ARM aarch64
$ file hello.ko
hello.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=8f24215154dff2b07306d655040f7c29de7a5efb, with debug_info, not stripped

$ scp hello.ko tx2@192.168.6.157:~

$ ssh tx2@192.168.6.157

# TX2
$ sudo insmod hello.ko
$ sudo rmmod hello
$ dmesg
...
[26332.410359] hello: no symbol version for module_layout
[26332.416120] hello: loading out-of-tree module taints kernel.
[26332.425103] Hello Module Test
[26332.428150] Hello Module Test2
[26347.063612] Goodbye Hello

As shown in the figure

Longterm Kernel

The Linux Kernel Archives - Releases , Jetson's current version 4.9 was release d at the end of 16 and supported until the beginning of 23

4.9 online documentation reference of kernel: Linux Kernel Documentation — The Linux Kernel documentation , a little old... It looks comfortable without the new 5.1x organization

Command memo

# Display loaded modules
lsmod
# View kernel module information
modinfo
# Load kernel module
indmod
# Uninstall kernel module
rmmod
# View module dependencies
depmod
# Load or remove modules
# It is more intelligent and can recursively resolve module dependencies
modprobe

# Determine the file type, 32/64bit
file

reference resources

Welcome to scan the two-dimensional code, and pay attention to WeChat official account.

Topics: kernel nvidia Jetson Module

Hot Topics

©2024 Programmer Think