Embedded Linux Container Technology

I. Linux Container Technology

1. The Linux Container container is a kernel virtualization technology that provides lightweight virtualization to isolate processes and resources.

2. LXC is short for Linux Container. Lightweight virtualization can be provided to isolate processes and resources without the need for instruction interpretation mechanisms and other complexities of full virtualization. It is equivalent to NameSpace in C++. Containers effectively divide resources managed by a single operating system into isolated groups to better balance conflicting resource usage requirements among isolated groups. Compared with traditional virtualization technology, its advantages lie in:

   (1) Using the same kernel as the host, the performance loss is small.

   (2) No instruction level simulation is required;

   (3) Just-in-time compilation is not required;

   (4) Containers can run instructions locally in the CPU core without any special interpretation mechanism.

   (5) It avoids the complexity of quasi-virtualization and system call substitution.

   (6) Lightweight isolation, while isolating, provides a sharing mechanism to achieve resource sharing between container and host.

   Summary: Linux Container is a lightweight virtualization tool.

3. Linux Container provides a mechanism to support simultaneous execution of multiple isolated server container s on a single controllable host node. Linux Container is a bit like chroot, providing a virtual environment with its own processes and network space, but different from virtual machines, because lxc is a virtualization of resources at the operating system level.

4. The relationship between LXC and docker

(1) Docker is not an alternative to LXC. The bottom layer of Docker is implemented using LXC. LXC sandboxes Linux processes, isolating them from each other and controlling them.
Resource allocation for processes.

(2) On the basis of LXC, Docker provides a series of more powerful functions.

2. Embedded Linux Container

The mainstream containers of Linux include LXC and Docker, among which LXC is the native lightweight container technology of Linux. Docker is implemented based on LXC, but Docker's main application scenario is cloud services.
It has powerful functions, but its support for embedded Linux development platform may not be so friendly. balenaOS claims to streamline Docker so it can run reliably
In embedded Linux environment. It is based on Yocto framework, and system D is used as init system. Compared with LXC, LXC is lighter and less demanding for system resources. Therefore, LXC is natural.
It is suitable for embedded Linux environment. However, (balenaOS)docker and LXC have their own merits, which require in-depth experience before reaching the final conclusion.

3. Experience Container

3.1 LXC

3.1.1 Hardware Environment

• allwinner T3
• Cortex-A7 quadruplex
• 1G RAM
• Ubuntu Core 18.04(Linux kernel 3.10.108)

3.1.2 LXC Installation

System requirements:

• Hard requirements:

  1. A C Library in glibc, libc and uclib
  2. Linux kernel >= 2.6.32

• lxc-attach additional requirements:

  1. Linux kernel >= 3.8

• Additional requirements for non-privileged containers:

  1. libpam-cgfs, cgmanager or other CGroup managers configure systems for non-privileged CGroups operations
  2. The latest version of shadow, including newuidmap and newgidmap
  3. Linux kernel >= 3.12

• Recommendation libraries:

  1. libcap (to allow for capability drops)
  2. libapparmor (to set a different apparmor profile for the container)
  3. libselinux (to set a different selinux context for the container)
  4. libseccomp (to set a seccomp policy for the container)
  5. libgnutls (for various checksumming)
  6. liblua (for the LUA binding)
  7. python3-dev (for the python3 binding)

In Ubuntu environment, the installation of LXC is very simple, and only one command can complete the installation of LXC, including all the library files it depends on:

	$sudo  apt-get  install  lxc

3.1.3 Create Unprivileged Containers

Unprivileged containers are the safest containers. Those that use uid and gid mappings assign a series of UIDs and gids to containers. This means that uid 0 (root) in the container is actually like uid 100000 outside the container. Therefore, if problems arise and an attacker manages to escape the container, they will find that they have as many rights as unmanned users.

Unfortunately, this also means that the following common operations are not allowed:

• Install most file systems
• Create device nodes
• Any operation on uid / gid outside of the mapping set

Note: Because the kernel version of the Ubuntu Core environment tested is 3.10.108, the system does not support non-privileged containers.

3.1.4 Create Privilege Containers

** Note: ** The container created below is a privileged container, that is, the container is created and started through root.

After the LxC installation is complete, you can use lxc-checkconfig to detect whether the Linux kernel supports containers.

	$ lxc-checkconfig

	--- Namespaces ---
	Namespaces: enabled
	Utsname namespace: enabled
	Ipc namespace: enabled
	Pid namespace: enabled
	**User namespace: missing**
	Network namespace: enabled
	Multiple /dev/pts instances: enabled
	
	--- Control groups ---
	Cgroups: enabled
	
	Cgroup v1 mount points: 
	/sys/fs/cgroup/systemd
	/sys/fs/cgroup/blkio
	/sys/fs/cgroup/cpuset
	/sys/fs/cgroup/devices
	/sys/fs/cgroup/perf_event
	/sys/fs/cgroup/cpu,cpuacct
	/sys/fs/cgroup/memory
	/sys/fs/cgroup/freezer
	
	Cgroup v2 mount points: 
	
	
	Cgroup v1 clone_children flag: enabled
	Cgroup device: enabled
	Cgroup sched: enabled
	Cgroup cpu account: enabled
	Cgroup memory controller: enabled
	Cgroup cpuset: enabled
	
	--- Misc ---
	Veth pair device: enabled, loaded
	Macvlan: enabled, not loaded
	Vlan: enabled, not loaded
	Bridges: enabled, loaded
	Advanced netfilter: enabled, not loaded
	CONFIG_NF_NAT_IPV4: enabled, loaded
	CONFIG_NF_NAT_IPV6: enabled, not loaded
	CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
	CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
	CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
	CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
	FUSE (for use with lxcfs): enabled, not loaded
	
	--- Checkpoint/Restore ---
	checkpoint restore: enabled
	CONFIG_FHANDLE: enabled
	CONFIG_EVENTFD: enabled
	CONFIG_EPOLL: enabled
	CONFIG_UNIX_DIAG: enabled
	CONFIG_INET_DIAG: enabled
	CONFIG_PACKET_DIAG: enabled
	CONFIG_NETLINK_DIAG: enabled
	File capabilities: 
	
	Note : Before booting a new kernel, you can check its configuration
	usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

This command detects the kernel's support for containers based on the. config file of the kernel.

lxc-create is used to create a container. lxc provides many templates when creating a container. The location of these templates is: / usr/share/lxc/templates / directory.

	sudo lxc-busybox  lxc-download  lxc-local  lxc-oci

Next, through lxc-download, create a distribution for Ubuntu, release for bionic, and arch for armhf container:

	sudo lxc-create -n c1 -t download

The above command is used to create a container named c1. When you type the command, it prompts you to enter parameters such as system, system version, architecture, etc. After that, lxc downloads rootfs of the corresponding container. Rootfs may take a long time to download. After the download is successful, the container has been downloaded successfully. The directory of rootfs is located in: / var/lib/lxc/c1, which includes a config configuration file and rootfs system.

Through lxc-ls, we can see that the C1 container has been deployed in the system, and lxc-info can view the current running status of c1.

Lxc-start is used to start a container. Specific parameters can be viewed through lxc-start-h.

    sudo lxc-start -n c1 -o logfile -d

problem

1.apparmor configuration problem

	  1 lxc-start c1 20190812055509.777 ERROR    apparmor - lsm/apparmor.c:apparmor_process_label_set:223 - If you really want to start this container, set
	  2 lxc-start c1 20190812055509.812 ERROR    apparmor - lsm/apparmor.c:apparmor_process_label_set:224 -lxc.apparmor.allow_incomplete = 1
	  3 lxc-start c1 20190812055509.822 ERROR    apparmor - lsm/apparmor.c:apparmor_process_label_set:225 - in your container configuration file
	  4 lxc-start c1 20190812055509.842 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
	  5 lxc-start c1 20190812055509.908 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
	  6 lxc-start c1 20190812055509.953 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
	  7 lxc-start c1 20190812055509.970 ERROR    lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
	  8 lxc-start c1 20190812055509.984 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
	  9 lxc-start c1 20190812055509.577 ERROR    start - start.c:__lxc_start:1939 - Failed to spawn container "c1"

By analyzing the log, we know that we need to add a "lxc.apparmor.allow_incomplete = 1" in the container configuration file. The configuration file of the c1 container is / var/lib/lxc/c1/config. Add this configuration to the end of config.

After completing the above steps, the container c1 is restarted, but it is still not started successfully. The error log is:

	  1 lxc-start c1 20190812034836.659 ERROR    lsm - lsm/lsm.c:lsm_process_label_set_at:174 - No such file or directory - Failed to set AppArmor label "lxc-container-default"
	  2 lxc-start c1 20190812034836.659 ERROR    apparmor - lsm/apparmor.c:apparmor_process_label_set:245 - Failed to change apparmor profile to lxc-container-default
	  3 lxc-start c1 20190812034836.660 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
	  4 lxc-start c1 20190812034836.660 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
	  5 lxc-start c1 20190812034836.661 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
	  6 lxc-start c1 20190812034836.662 ERROR    lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
	  7 lxc-start c1 20190812034836.662 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
	  8 lxc-start c1 20190812034836.719 ERROR    start - start.c:__lxc_start:1939 - Failed to spawn container "c1"

Analyzing logs, AppArmor could not find the "lxc-container-default" configuration file. AppArmor is a security module similar to SeLinux provided by the Linux kernel. Ubuntu defaults
Support this function. However, Ubuntu Core does not support apparmor_status and apparmor_parser tools by default, and there are few profile s in the system, so you can
By installing through sudo apt-get install apparmor-profiles, the apparmor_status and apparmor_parser tools above will also be installed together.

After the installation of apparmor profile is completed, it is necessary to restart apparmor. The starting method is as follows:

	Start : sudo /etc/init.d/apparmor start
	Stop : sudo /etc/init.d/apparmor stop
	reload: sudo /etc/init.d/apparmor reload
	Show status: sudo /etc/init.d/apparmor status

Through sudo lxc-info-n c1, you can see the operation status of the container:

	sudo lxc-info -n c1
	
	Name:           c1
	State:          RUNNING
	PID:            2993
	IP:             10.0.3.81
	CPU use:        1.87 seconds
	BlkIO use:      18.83 MiB
	Memory use:     22.02 MiB
	KMem use:       0 bytes
	Link:           vethLWC02H
	 TX bytes:      1.04 KiB
	 RX bytes:      1.12 KiB
	 Total bytes:   2.16 KiB

You can see the current system resource utilization of the current c1 container.

With the lxc-attach command, you can get a shell that interacts with the container, such as

   sudo lxc-attach -n c1

When attach succeeds, it enters a shell terminal.

	#root@c1:

Any command in the container can be executed through lxc-attach, such as:

	sudo lxc-attach -n c1 ls
	sudo lxc-attach -n c1 top

APP can be packaged as deb, then DEB is copied to the appropriate location of the container rootfs, and finally DEB is installed by executing the lxc-attach command.

	sudo lxc-attach -n c1 -- dpkg -i /home/ubuntu/app.deb  //Install app.deb
	sudo lxc-attach -n c1 -- dpkg  -r /home/ubuntu/app.deb //Unload app.deb

The terminal provides almost all of Ubuntu's functions, such as apt, but the Ubuntu runs in a container mode with the Ubuntu Core physical host.

Finally, lxc-stop closes the container, and lxc-destroy deletes all container-related configuration files, including rootfs.

3.2 LXD

To be continued...

3.3 BalenaOS

To be continued...