Recently, the official website of CNCF(Cloud Native Computing Foundation) published an article jointly written by Jin Chen, an R & D Intern of harmonic cloud edge computing, and Vivian Hu from Second State - "using OpenYurt to manage WasmEdge|lightweight edge computing solution".
This paper introduces how the edge computing framework OpenYurt uses lightweight runtime WasmEdge at the edge. Among them, Jin Chen, an intern in research and development of harmonic cloud edge computing, is mainly responsible for the experimental operation, including experimental recording. In the actual scenario, docker is too heavy as runtime in the scenario with limited edge resources. In this experiment, containerd is used as high level runtime, and the corresponding containerd configuration is modified to make the default low level runtime of containerd change from runc to crun. Finally, the demo app based on web assembly is successfully run on OpenYurt cluster. Finally, through the integration of OpenYurt and WasmEdge, we can broaden the use of edge computing in the scene of limited edge resources. Jin Chen said: "this article is just a preliminary attempt of cooperation between the two sides. In the future, more cooperation possibilities will be developed for specific scenes."
Harmonic cloud edge computing solution ensures stability through EdgeStack intelligent edge computing platform, and EdgeBox edge all-in-one machine ensures edge security. Among them, EdgeStack ® The intelligent edge computing platform is based on the lightweight container arrangement framework cloud edge end collaboration technology to create a "cloud + edge + end" cloud edge end collaboration architecture, sink the container cloud computing capacity to the edge nodes, and support the access of millions of edge nodes and devices. The intelligent edge computing platform supports KubeEdge and OpenYurt at the same time, and completes massive data edge processing through lightweight edge components in collaboration with the cloud; Support the automatic distribution, update and operation and maintenance of edge applications and models from the cloud, and greatly reduce the pressure of edge applications and data operation and maintenance management.
HC-EdgeStack ® Architecture diagram of edge computing platform
Using OpenYurt to manage WasmEdge | lightweight edge computing solution
It is an architecture that places the data of [Vivian] computing resources close to the edge of [Vivian], for example. In this way, it provides low latency, high performance and strong security for various applications. However, similar to cloud computing in data center, edge computing also faces challenges such as workload distribution, management and scheduling across heterogeneous devices and networks. In order to meet these challenges in cloud computing, many developers will choose container based solutions. Therefore, developers are now introducing solutions such as Kubernetes (K8s) and containers into edge computing. Unfortunately, neither K8s nor Linux containers are designed for edge computing. Potential problems with standard K8s on the edge include:
- The computing, storage and network resources of edge devices are limited. Running a complete K8s node on an edge device is often too resource intensive.
- K8s can only manage devices with public IP addresses. However, many edge devices do not expose their public IP.
- Edge devices are often offline, while the standard K8s requires their pod s to be always connected.
- Edge devices are usually connected through non TCP networks and require a special network interface to communicate with their K8s manager.
Therefore, developers have created a special Kubernetes distribution for edge computing. A leading K8s release optimized for edge is OpenYurt[2]. OpenYurt was originally created by Alibaba and is now a sandbox project of CNCF. OpenYurt is an open platform that includes the powerful editing capabilities of upstream Kubernetes. Therefore, users can use the same upstream Kubernetes API to deliver, manage, and monitor edge workloads.
Even with edge optimized k8s distributions like OpenYurt, the standard Linux container is still too bulky to run edge workloads on small devices. A typical Linux container image usually takes up space in gigabytes, and it takes a few seconds to start a full CPU. Considering that typical edge applications are usually thin, Linux containers occupy too much storage, CPU and battery resources on resource constrained devices. In addition, Linux container applications are not cross platform and unsafe on edge devices. For edge containerized applications, OCI compliant and k8s compatible [3] lightweight web assembly runtime, such as WasmEdge[4], is a better alternative than Linux container [5]. WasmEdge provides a capability based security sandbox, makes few assumptions about the underlying operating system services, and supports multiple programming languages at the front end [6]. In addition to Linux, macOS and Windows, WasmEdge can even run on real-time operating system (RTOS), such as seL4 of Linux foundation [7].
OpenYurt[8] and WasmEdge[9] team showed how to use OpenYurt to manage WasmEdge applications in an edge cluster. OpenYurt and WasmEdge will finally bring cloud native container applications to edge computing! Let's start!
preparation
OpenYurt uses the centralized Kubernetes control panel to manage multiple edge nodes in the ECS. In addition, it extends the upstream Kubernetes to the edge. In this demonstration, we will set up two machines. One simulated cloud Node is called Master, and the other simulated edge Node is called Node. These two nodes constitute the simplest OpenYurt cluster in which OpenYurt components run. First, close the swap partition
sudo swapoff -a free -m //verification
Next, configure the file node/etc/hosts of the two nodes
Then, load the kernel module br_netfilter and adjust the kernel parameters.
//load the module sudo modprobe br_netfilter //verify lsmod | grep br_netfilter // create k8s.conf cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sudo sysctl --system
Finally, set the RP filter value and set the file / etc / sysctl d/10-network-security. The values of the two parameters in conf are changed from 2 to 1, and / proc/sys/net/ipv4/ip_forward is set to 1.
sudo vi /etc/sysctl.d/10-network-security.confecho 1 > /proc/sys/net/ipv4/ip_forward sudo sysctl --system
Install container
Use the following command line to install containerd on the edge node.
export VERSION="1.5.7"
echo -e "Version: $VERSION"
echo -e "Installing libseccomp2 ..."
sudo apt install -y libseccomp2
echo -e "Installing wget"
sudo apt install -y wget
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz.sha256sum sha256sum --check cri-containerd-cni-${VERSION}-linux-amd64.tar.gz.sha256sum
sudo tar --no-overwrite-dir -C / -xzf cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
sudo systemctl daemon-reloadSince the crun project supports WasmEdge by default, we only need to configure containerd for runc. So we need to modify / etc / containerd / config The runc parameter in toml is curr, and add pod_annotation.
sudo mkdir -p /etc/containerd/ sudo bash -c "containerd config default > /etc/containerd/config.toml" wget https://raw.githubusercontent.com/second-state/wasmedge-containers-examples/main/containerd/containerd_config.diff sudo patch -d/ -p0 < containerd_config.diff
After that, restart containerd to make the configuration take effect.
systemctl start containerd
Install WasmEdge
Use the following installation script to install WasmEdge on the edge node.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash
Install and build crun
We need to build a crun binary file supporting WasmEdge on edge node. At present, the most direct way is to build crun from the source code. First, make sure you have installed crun dependencies in Ubuntu 20.04. If you are using other versions of Linux, please refer to the official document [10].
sudo apt update
sudo apt install -y make git gcc build-essential pkgconf libtool \
libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev \
go-md2man libtool autoconf python3 automakeNext, configure, build, and install crun binaries with WasmEdge support
sudo make install git clone https://github.com/containers/crun cd crun ./autogen.sh ./configure —with-wasmedge make sudo make install
Install OpenYurt
Now, we can install OpenYurt on the Edge node to control the containerd, crun and wasedge stacks under it. The easiest way to build an OpenYurt cluster is to use the OpenYurt experience center. You just need to register an account for testing, and then you can get an OpenYurt cluster. Next, you can use the yurtctl join command line to join Edge nodes. See more details here [11]. At present, the OpenYurt component version provided by the OpenYurt experience center is 0.6.0 (the latest version), and the Kubernetes version is 1.20.11.
Alternatively, you can build OpenYurt clusters from scratch according to the following guidelines.
Set up a K8s cluster
Here, we use yurtctl convert to convert K8s cluster into OpenYurt cluster, so we need to build a K8s cluster first. It should be noted that if you use yurtctl init/join to set up OpenYurt cluster and join nodes, you can skip this step.
$ sudo apt-get update && sudo apt-get install -y ca-certificates curl software-properties-common apt-transport-https // add K8s source $ curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - $ sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF $ deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main // install K8s components 1.18.9 $ sudo apt-get update && sudo apt-get install -y kubelet=1.18.9-00 kubeadm=1.18.9-00 kubectl=1.18.9-00 // Initialize the master node $ sudo kubeadm init --pod-network-cidr 172.16.0.0/16 \ --apiserver-advertise-address=192.168.3.167 \ --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers // join the work node $ kubeadm join 192.168.3.167:6443 --token 3zefbt.99e6denc1cxpk9fg \ --discovery-token-ca-cert-hash sha256:8077d4e7dd6eee64a999d56866ae4336073ed5ffc3f23281d757276b08b9b195
Install yurtctl tool
Use the following command line to install yurtctl. The yurtcli CLI tool can install / uninstall OpenYurt and convert a standard Kubernetes cluster to an OpenYurt cluster.
git clone https://github.com/openyurtio/openyurt.git cd openyurt make build WHAT=cmd/yurtctl
Install OpenYurt components
yurtctl convert --deploy-yurttunnel --cloud-nodes oy-master --provider kubeadm\ --yurt-controller-manager-image="openyurt/yurt-controller-manager:v0.5.0"\ --yurt-tunnel-agent-image="openyurt/yurt-tunnel-agent:v0.5.0"\ --yurt-tunnel-server-image="openyurt/yurt-tunnel-server:v0.5.0"\ --node-servant-image="openyurt/node-servant:latest"\ --yurthub-image="openyurt/yurthub:v0.5.0"
Note: it is necessary to change -- yurtctl service image = "openyurt / yurtctl service: v0.5.0" to -- node service image = "openyurt / node service: latest"
Run a simple WebAssembly app
Next, let's run the web assembly program through the Kubernetes cluster as a container in the pod. This section will start with Docker hub pulling the container image based on web assembly. If you want to know how to compile and package the WebAssembly program and publish it as a container image to the Docker hub, please refer to WasmEdge Book[12].
Enter the following command from another terminal window of the master node, and then start using the cluster.
root@master:~# kubectl cluster-info Kubernetes master is running at https://192.168.3.167:6443 KubeDNS is running at https://192.168.3.167:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Because kubectl 1.18.9 has no annotations parameter, we need to adjust the command line here. If you are using OpenYurt Experience Center and its default versions are OpenYurt 0.6.0 and Kubernetes 1.20.11, please directly refer to Kubernetes Section [13] in WasmEdge Book to run wasm application.
// kubectl 1.18.9
$ sudo kubectl run -it --rm --restart=Never wasi-demo --image=hydai/wasm-wasi-example:with-wasm-annotation --overrides='{"kind":"Pod","metadata":{"annotations":{"module.wasm.image/variant":"compat"}} , "apiVersion":"v1", "spec": {"hostNetwork": true}}' /wasi_example_main.wasm 50000000
// kubectl 1.20.11
$ sudo kubectl run -it --rm --restart=Never wasi-demo --image=hydai/wasm-wasi-example:with-wasm-annotation --annotations="module.wasm.image/variant=compat" --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork": true}}' /wasi_example_main.wasm 50000000The output of the container application is printed to the console. All Kubernetes versions are the same.
Random number: 1123434661 Random bytes: [25, 169, 202, 211, 22, 29, 128, 133, 168, 185, 114, 161, 48, 154, 56, 54, 99, 5, 229, 161, 225, 47, 85, 133, 90, 61, 156, 86, 3, 14, 10, 69, 185, 225, 226, 181, 141, 67, 44, 121, 157, 98, 247, 148, 201, 248, 236, 190, 217, 245, 131, 68, 124, 28, 193, 143, 215, 32, 184, 50, 71, 92, 148, 35, 180, 112, 125, 12, 152, 111, 32, 30, 86, 15, 107, 225, 39, 30, 178, 215, 182, 113, 216, 137, 98, 189, 72, 68, 107, 246, 108, 210, 148, 191, 28, 40, 233, 200, 222, 132, 247, 207, 239, 32, 79, 238, 18, 62, 67, 114, 186, 6, 212, 215, 31, 13, 53, 138, 97, 169, 28, 183, 235, 221, 218, 81, 84, 235] Printed from wasi: This is from a main function This is from a main function The env vars are as follows. The args are as follows. /wasi_example_main.wasm 50000000 File content is This is in a file pod "wasi-demo" deleted
Now you can view the status of the pod from the Kubernetes command line.
crictl ps -a
You can see the events from scheduling to running the web assembly workload in the log.
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID 0c176ed65599a 0423b8eb71e31 8 seconds ago Exited wasi-demo
That's it. Next, you can try to use OpenYurt to manage your WasmEdge application.
Future expectation
WasmEdge[14] and OpenYurt[15] jointly introduced all the advantages of Kubernetes into edge computing. The above demo demonstrates a very simple usage scenario, which only touches on the corner of its great potential and prospect. Please follow these two projects on GitHub to get the latest information about our cooperation. At the same time, if you have applications that benefit from this technology path, we are also happy to listen!
- WasmEdge GitHub repo: https://github.com/WasmEdge/WasmEdge[16]
- OpenYurt GitHub repo: https://github.com/openyurtio/openyurt[17]
reference material
[1] Edge calculation: https://en.wikipedia.org/wiki/Edge_computing [2] OpenYurt: https://openyurt.io/ [3] OCI standard and compatible k8s with: https://wasmedge.org/book/en/kubernetes.html [4] WasmEdge: https://wasmedge.org/ [5] Compared to Linux containers: https://wasmedge.org/wasm_linux_container/ [6] Multiple programming languages: https://wasmedge.org/book/en/dev.html [7] seL4: https://wasmedge.org/book/en/os/sel4.html [8] OpenYurt: https://github.com/openyurtio/openyurt [9] WasmEdge: https://github.com/WasmEdge/WasmEdge [10] Official documents: https://github.com/containers/crun#readme [11] Here: https://openyurt.io/docs/installation/openyurt-experience-center/overview/ [12]WasmEdge Book: https://wasmedge.org/book/en/
kubernetes/demo/wasi.html [13]Kubernetes section: https://wasmedge.org/book/en/
kubernetes/kubernetes.html
[14] WasmEdge: https://github.com/WasmEdge/WasmEdge [15] OpenYurt: https://github.com/openyurtio/openyurt [16] https://github.com/WasmEdge/WasmEdge: https://github.com/WasmEdge/WasmEdge [17] https://github.com/openyurtio/openyurt: https://github.com/openyurtio/openyurt
About WasmEdge
WasmEdge is a lightweight, secure, high-performance, real-time software container and running environment. At present, it is a CNCF sandbox project. WasmEdge is applied in SaaS, cloud native, service mesh, edge computing, automobile and other fields.