Running logic of kubebuilder

summary

The following is kubebuilder's Architecture diagram . You can see that the outermost layer is driven by a component called Manager, which contains multiple components. The mapping relationship between gvk and informer is saved in the Cache, which is used to Cache kubernetes objects through informer. The Controller uses the workqueue method to Cache the objects passed by the informer, and then extracts the objects in the workqueue and passes them to the Reconciler for processing.

This article does not introduce the usage of kuberbuilder. If necessary, you can refer to the following three articles:

• Kubernetes Operator for Beginners — What, Why, How
• Advanced Kubernetes Operators Development
• Advanced Kubernetes Operator Development with Finalizer, Informer, and Webhook

The version of controller runtime used this time is: v0 eleven

Code generation reference for the following example: Building your own kubernetes CRDs

Managers

manager Be responsible for running controllers and webhooks and setting public dependencies, such as clients, caches, schemes, etc.

Processing of kubebuilder

kubebuilder will be automatically in main To create a Manager in go:

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:                 scheme,
		MetricsBindAddress:     metricsAddr,
		Port:                   9443,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "3b9f5c61.com.bolingcavalry",
	})

controllers is by calling manager The start interface starts.

Controllers

controller Use events to trigger the request for reconcile. Through controller The new interface can initialize a controller through manager Start starts the controller.

func New(name string, mgr manager.Manager, options Options) (Controller, error) {
	c, err := NewUnmanaged(name, mgr, options)
	if err != nil {
		return nil, err
	}

	// Add the controller as a Manager components
	return c, mgr.Add(c) // Add controller to manager
}

Processing of kubebuilder

kubebuilder will be automatically in main Go generates a SetupWithManager function, creates it in Complete and adds the controller to the manager, as shown below:

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Guestbook{}).
		Complete(r)
}

In main Calling Manager. in go Start the controller interface:

mgr.Start(ctrl.SetupSignalHandler())

Reconcilers

The core of the Controller is to implement the Reconciler interface. Reconciler You will receive a reconcile request that contains the name and namespace of the object. Reconcile will compare the current state and expected state of the object and the resources it owns, and make corresponding adjustments accordingly.

Generally, the Controller will trigger reconcile based on cluster events (such as Creating, Updating, Deleting Kubernetes objects) or external events (such as GitHub Webhooks, polling external resources, etc.).

Note: the reqeust passed in from recomciler only contains the name and namespace of the object, and there is no other information about the object. Therefore, you need to obtain the relevant information of the object through kubernetes client.

type Request struct {
	// NamespacedName is the name and namespace of the object to reconcile.
	types.NamespacedName
}

type NamespacedName struct {
	Namespace string
	Name      string
}

The Reconciler interface is described as follows, in which an example of its processing logic is given:

• Read an object and all its pod s
• It is observed that the expected number of copies of the object is 5, but there is only one pod copy
• Create 4 pods and set OwnerReferences

/*
Reconciler implements a Kubernetes API for a specific Resource by Creating, Updating or Deleting Kubernetes
objects, or by making changes to systems external to the cluster (e.g. cloudproviders, github, etc).

reconcile implementations compare the state specified in an object by a user against the actual cluster state,
and then perform operations to make the actual cluster state reflect the state specified by the user.

Typically, reconcile is triggered by a Controller in response to cluster Events (e.g. Creating, Updating,
Deleting Kubernetes objects) or external Events (GitHub Webhooks, polling external sources, etc).

Example reconcile Logic:

	* Read an object and all the Pods it owns.
	* Observe that the object spec specifies 5 replicas but actual cluster contains only 1 Pod replica.
	* Create 4 Pods and set their OwnerReferences to the object.

reconcile may be implemented as either a type:

	type reconcile struct {}

	func (reconcile) reconcile(controller.Request) (controller.Result, error) {
		// Implement business logic of reading and writing objects here
		return controller.Result{}, nil
	}

Or as a function:

	controller.Func(func(o controller.Request) (controller.Result, error) {
		// Implement business logic of reading and writing objects here
		return controller.Result{}, nil
	})

Reconciliation is level-based, meaning action isn't driven off changes in individual Events, but instead is
driven by actual cluster state read from the apiserver or a local cache.
For example if responding to a Pod Delete Event, the Request won't contain that a Pod was deleted,
instead the reconcile function observes this when reading the cluster state and seeing the Pod as missing.
*/
type Reconciler interface {
	// Reconcile performs a full reconciliation for the object referred to by the Request.
	// The Controller will requeue the Request to be processed again if an error is non-nil or
	// Result.Requeue is true, otherwise upon completion it will remove the work from the queue.
	Reconcile(context.Context, Request) (Result, error)
}

Processing of kubebuilder

kubebuilder will be in guestbook_controller.go generates a template that implements the Reconciler interface:

func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

So how does reconciler relate to controller? As mentioned above, kubebuilder will create and add controller to manager through Complete (called in SetupWithManager). At the same time, we can see reconcile. is introduced in Complete. Reconciler interface, which is the entry associated with controller and reconciler:

func (blder *Builder) Complete(r reconcile.Reconciler) error {
	_, err := blder.Build(r)
	return err
}

Follow up meeting: Builder Build -->Builder. Docontroller -- > newcontroller is finally passed to the controller's initialization interface controller New and assign it to controller Do variable. controller. The controller structure created in new is as follows. It can be seen that MakeQueue is also given a function to create workqueue. New events will be cached in the workqueue and then passed to Reconcile for processing:

	// Create controller with dependencies set
	return &controller.Controller{
		Do: options.Reconciler,
		MakeQueue: func() workqueue.RateLimitingInterface {
			return workqueue.NewNamedRateLimitingQueue(options.RateLimiter, name)
		},
		MaxConcurrentReconciles: options.MaxConcurrentReconciles,
		CacheSyncTimeout:        options.CacheSyncTimeout,
		SetFields:               mgr.SetFields,
		Name:                    name,
		Log:                     options.Log.WithName("controller").WithName(name),
		RecoverPanic:            options.RecoverPanic,
	}, nil

As mentioned above, the controller will call Reconciler according to events. How does it pass events?

You can see the start interface of the Controller (the Controller.Start interface will be called in Manager.Start), and you can see that it calls processNextWorkItem to handle events in workqueue:

func (c *Controller) Start(ctx context.Context) error {
	...

	c.Queue = c.MakeQueue() //Initialize a workqueue through MakeQueue
	...

	wg := &sync.WaitGroup{}
	err := func() error {
        ...
		wg.Add(c.MaxConcurrentReconciles)
		for i := 0; i < c.MaxConcurrentReconciles; i++ {
			go func() {
				defer wg.Done()
				for c.processNextWorkItem(ctx) {
				}
			}()
		}
		...
	}()
	...
}

Continue to check processNextWorkItem. You can see that the processing logic is the same as that of workqueue in client go. Take out the event object from workqueue and pass it to reconcileHandler:

func (c *Controller) processNextWorkItem(ctx context.Context) bool {
	obj, shutdown := c.Queue.Get() //Get the object in workqueue
	if shutdown {
		// Stop working
		return false
	}

	defer c.Queue.Done(obj)

	ctrlmetrics.ActiveWorkers.WithLabelValues(c.Name).Add(1)
	defer ctrlmetrics.ActiveWorkers.WithLabelValues(c.Name).Add(-1)

	c.reconcileHandler(ctx, obj)
	return true
}

Later, it will pass the controller reconcileHandler --> Controller. Reconcile -->Controller. Do. Reconcile finally passes the event to reconcile (the reconcile implemented by itself is assigned to the do variable of the controller).

To sum up: kubebuilder first assigns Reconcile to controller through SetupWithManager, and will call controller when the Manager starts Start starts the controller. The controller will continuously obtain the objects in its workqueue and pass them to Reconcile for processing.

Controller event source

The above describes how the controller handles events. How do the events in the workqueue come from?

Back to builder Complete-->Builder. Build. From the above content, we can know that the controller is initialized in the doController function, and Reconciler and controller are associated. There is a doWatch function below, which registers the object type to be watched and EventHandler (type: handler.EnqueueRequestForObject), and starts the monitoring of resources through the controller's watch interface:

func (blder *Builder) Build(r reconcile.Reconciler) (controller.Controller, error) {
	...
	// Set the ControllerManagedBy
	if err := blder.doController(r); err != nil {//Initialize controller
		return nil, err
	}

	// Set the Watch
	if err := blder.doWatch(); err != nil {
		return nil, err
	}

	return blder.ctrl, nil
}

func (blder *Builder) doWatch() error {
	// Reconcile type
	typeForSrc, err := blder.project(blder.forInput.object, blder.forInput.objectProjection)//Format resource type
	if err != nil {
		return err
	}
	src := &source.Kind{Type: typeForSrc} //Initialize resource type
	hdler := &handler.EnqueueRequestForObject{} //Initialize eventHandler
	allPredicates := append(blder.globalPredicates, blder.forInput.predicates...)
	if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { //Start monitoring of resources
		return err
	}
    ...
}

Blder forInput. Object is the parameter of For in SetupWithManager (& webappv1. Guestbook {})

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Guestbook{}).
		Complete(r)
}

Keep looking at controller Watch interface, you can see that it calls Src Associate the parameters of & Kindle. Qeuc {and & requester. Qeuc} with the type of Kindle. Qeuc}

func (c *Controller) Watch(src source.Source, evthdler handler.EventHandler, prct ...predicate.Predicate) error {
   ...
   return src.Start(c.ctx, evthdler, c.Queue, prct...)
}

At kind According to KS Type select the appropriate informer and add the event manager internal EventHandler:

When the Manager is initialized (if not specified), a Cache will be created by default, and gvk is saved in the Cache Mapping relationship of sharedindexinformer, KS Cache. Getinformer will extract the gvk information of the object and obtain the informer according to the gvk.

In manager When starting, the informer in the Cache will be started.

func (ks *Kind) Start(ctx context.Context, handler handler.EventHandler, queue workqueue.RateLimitingInterface,
	prct ...predicate.Predicate) error {
	...
	go func() {
		...
		if err := wait.PollImmediateUntilWithContext(ctx, 10*time.Second, func(ctx context.Context) (bool, error) {
			// Lookup the Informer from the Cache and add an EventHandler which populates the Queue
			i, lastErr = ks.cache.GetInformer(ctx, ks.Type)
			...
			return true, nil
		}); 
        ...
		i.AddEventHandler(internal.EventHandler{Queue: queue, EventHandler: handler, Predicates: prct})
		...
	}()

	return nil
}

internal. The ResourceEventHandler interface required by SharedIndexInformer is implemented in EventHandler

type ResourceEventHandler interface {
	OnAdd(obj interface{})
	OnUpdate(oldObj, newObj interface{})
	OnDelete(obj interface{})
}

See how the EventHandler adds the object that OnAdd listens to to to the queue:

func (e EventHandler) OnAdd(obj interface{}) {
	...
	e.EventHandler.Create(c, e.Queue)
}

You can see in enqueuerequestforobject The name and namespace of the object are extracted from create and added to the queue:

func (e *EnqueueRequestForObject) Create(evt event.CreateEvent, q workqueue.RateLimitingInterface) {
	...
	q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
		Name:      evt.Object.GetName(),
		Namespace: evt.Object.GetNamespace(),
	}})
}

So far, the whole Kubebuilder is strung together.

Difference from using client go

client-go

When you need to operate kubernetes resources, you usually use client go to write the CRUD logic of resources, or use the informer mechanism to listen for resource changes, and handle them in OnAdd, OnUpdate and OnDelete.

kubebuilder Operator

From the above explanation, we can see that the Operator generally involves two aspects: object and all its (own) resources. Reconcilers are the core processing logic, but they can only get the name and namespace of the resource. They do not know what the operation (addition, deletion and modification) of the resource is or other information of the resource. The purpose is to adjust the state of the resource according to the expected state of the object when receiving the resource change.

kubebuilder also provides client Library, you can CRUD kubernetes resources, but it is recommended to directly use client go in this case:

package main

import (
	"context"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

var c client.Client

func main() {
	// Using a typed object.
	pod := &corev1.Pod{}
	// c is a created client.
	_ = c.Get(context.Background(), client.ObjectKey{
		Namespace: "namespace",
		Name:      "name",
	}, pod)

	// Using a unstructured object.
	u := &unstructured.Unstructured{}
	u.SetGroupVersionKind(schema.GroupVersionKind{
		Group:   "apps",
		Kind:    "Deployment",
		Version: "v1",
	})
	_ = c.Get(context.Background(), client.ObjectKey{
		Namespace: "namespace",
		Name:      "name",
	}, u)
}