kubernetes event reflector

Posted by Neumy on Mon, 06 May 2019 23:35:04 +0200

kubernetes event reflector

Reflector is an event reflector for Kubernetes. controller and Informer use it to change specific types of resource objects in List and Watch apiserver and update them to DeltaFIFO and object cache, so that clients can subsequently retrieve objects from local cache without having to interact with apiserver every time.

Before introducing Relfector, the ListerWatcher interface used by Reflector is introduced.

ListWatcher interface

The ListWatcher interface defines methods for List and Watch specific resource types:

  1. List(): Get a batch of specific types of objects from apiserver;
  2. Watch(): Starting with the ResourceVersion obtained from List() above, changes in objects are made through apiserver Watch etcd.

ListWatcher needs to communicate through the Get method of RESTClient and apiserver, which is defined by the Getter interface.

// From k8s.io/client-go/tools/cache/listwatch.go`
type ListerWatcher interface {
	// List() returns the list of objects; extracts ResourceVersion from these objects, the user's subsequent Watch() method parameters;
	List(options metav1.ListOptions) (runtime.Object, error)
	// The Watch() method starts Watch from the specified version (specified in options).
	Watch(options metav1.ListOptions) (watch.Interface, error)
}

// ListFunc knows how to list resources
type ListFunc func(options metav1.ListOptions) (runtime.Object, error)

// WatchFunc knows how to watch resources
type WatchFunc func(options metav1.ListOptions) (watch.Interface, error)

// Getter interface knows how to access Get method from RESTClient.
type Getter interface {
	Get() *restclient.Request
}

ListWatch Implementing ListWatcher Interface

The ListWatch type implements the ListWatcher interface and is used by the NewReflector() function and the Informer creation function of each K8S built-in resource object, such as NewFiltered Deployment Informer ():

// From k8s.io/client-go/tools/cache/listwatch.go
type ListWatch struct {
	ListFunc  ListFunc
	WatchFunc WatchFunc
	DisableChunking bool
}

The functions NewListWatchFromClient() and NewFilteredListWatchFromClient() return the ListWatch object.

The incoming Getter parameter is the REST Client of the K8S specific API Group/Version that has been configured. Here "configured" refers to rest.Config, which creates RESTClient, has been configured with parameters such as GroupVersion, APIPath, Negotiated Serializer, etc.

For K8S built-in objects, the configured REST Client is located in the k8s.io/client-go/kubernetes/<group>/<version>/<group>_client.go file, such as
ExtensionsV1beta1Client

For custom type objects, the configured RESTClient is located in the pkg/client/clientset/versioned/typed/<group>/<version>/<group>_client.go file.

This configured RESTClient is applicable to all resource types under a specific API Group/Version. When sending a Get() request, you also need to pass a specific resource type name to the Resource() method, such as Resource("deployments"), in order to uniquely determine the resource type: /<APIPath>/<group>/<version>/namespaces/<resource>/<name>, such as/apis/apps/v1beta1/namespaces/de Fault/deployment/my-nginx-111.

// From k8s.io/client-go/tools/cache/listwatch.go
func NewListWatchFromClient(c Getter, resource string, namespace string, fieldSelector fields.Selector) *ListWatch {
	optionsModifier := func(options *metav1.ListOptions) {
		options.FieldSelector = fieldSelector.String()
	}
	return NewFilteredListWatchFromClient(c, resource, namespace, optionsModifier)
}

// Resource type objects corresponding to List and Watch calling the Get() method of RESTClient
func NewFilteredListWatchFromClient(c Getter, resource string, namespace string, optionsModifier func(options *metav1.ListOptions)) *ListWatch {
	listFunc := func(options metav1.ListOptions) (runtime.Object, error) {
		optionsModifier(&options)
		// c is a REST Client for an API Group/Version
		return c.Get().
			Namespace(namespace).
			Resource(resource). // Specify a resource type name, such as "deployments"
			VersionedParams(&options, metav1.ParameterCodec).
			Do().
			Get()
	}
	// When the WatchFunc function is actually invoked, options contain the ResultVersion value of the list of objects that ListFunc put back last time
	watchFunc := func(options metav1.ListOptions) (watch.Interface, error) {
		options.Watch = true
		optionsModifier(&options)
		return c.Get().
			Namespace(namespace).
			Resource(resource).
			VersionedParams(&options, metav1.ParameterCodec).
			Watch()
	}
	return &ListWatch{ListFunc: listFunc, WatchFunc: watchFunc}
}

ListWatch's List() and Watch() methods are implemented by calling internal ListFunc() or WatchFunc() functions directly. They are more direct and simple, so they are no longer analyzed.

Informer using ListWatch

As we will see later, each resource type has its own specific Informer (codegen tool automatically generated), such as DeploymentInformer They initialize ListWatch using ClientSet of their own resource type and return only objects of the corresponding type:

// From k8s.io/client-go/informers/extensions/v1beta1/deployment.go
func NewFilteredDeploymentInformer(client kubernetes.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
	return cache.NewSharedIndexInformer(
		// Creating ListWatch with RESTClient of a specific resource type
		&cache.ListWatch{
			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
				if tweakListOptions != nil {
					tweakListOptions(&options)
				}
				return client.ExtensionsV1beta1().Deployments(namespace).List(options)
			},
			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
				if tweakListOptions != nil {
					tweakListOptions(&options)
				}
				return client.ExtensionsV1beta1().Deployments(namespace).Watch(options)
			},
		},
		&extensionsv1beta1.Deployment{},
		resyncPeriod,
		indexers,
	)
}

Reflector

After parsing the ListWatch pad, we can finally start parsing the Reflector implementation!

Reflector type definition

Reflector uses Lister Watcher to synchronize expectedType objects and events from apiserver and cache them into store (DeltaFIFO type).

// From: k8s.io/client-go/tools/cache/reflector.go
type Reflector struct {
	// The name of the Reflector and the default value is file:line
	name string
	
	// metrics tracks basic metric information about the reflector
	metrics *reflectorMetrics

	// A Reflector specifies the type of object to monitor, and the field specifies the type of object.
	expectedType reflect.Type
	
	// store is used to cache object change events, usually DeltaFIFO, with reference to the NewInformer/NewIndexerInformer function
	store Store

	// listerWatcher for List and Watch resource objects
	listerWatcher ListerWatcher

	// When the ListAndWatch() method returns with an error (timeout), wait for the period time to execute the ListAndWatch() method again
	period       time.Duration

	// The Reflector cycle calls the Reync () method of the store.
	// For DeltaFIFO, all objects in the knownObjects object cache are synchronized to DeltaFIFO
	resyncPeriod time.Duration
	
	// Additional Judgment Functions in Executing resync
	ShouldResync func() bool

	// clock allows tests to manipulate time
	clock clock.Clock
	
	// lastSyncResourceVersion is the resource version token last
	// observed when doing a sync with the underlying store
	// it is thread safe, but not synchronized with the underlying store
	lastSyncResourceVersion string
	
	// lastSyncResourceVersionMutex guards read/write access to lastSyncResourceVersion
	lastSyncResourceVersionMutex sync.RWMutex
}

Functions to create Reflector objects

The functions NewNamespace KeyedIndexerAndReflector (), NewReflector(), NewNamedReflector() return the Reflector object, where NewNamedReflector() is the basis of the other two methods:

// From: k8s.io/client-go/tools/cache/reflector.go
var internalPackages = []string{"client-go/tools/cache/"}

func NewNamespaceKeyedIndexerAndReflector(lw ListerWatcher, expectedType interface{}, resyncPeriod time.Duration) (indexer Indexer, reflector *Reflector) {
	// Use Namespace as IndexFunc and KeyFunc
	indexer = NewIndexer(MetaNamespaceKeyFunc, Indexers{"namespace": MetaNamespaceIndexFunc})
	reflector = NewReflector(lw, expectedType, indexer, resyncPeriod)
	return indexer, reflector
}

func NewReflector(lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector {
	// Create a Reflector name in file:linenum using the directory file where the Reflector resides
	return NewNamedReflector(naming.GetNameFromCallsite(internalPackages...), lw, expectedType, store, resyncPeriod)
}

func NewNamedReflector(name string, lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector {
	reflectorSuffix := atomic.AddInt64(&reflectorDisambiguator, 1)
	r := &Reflector{
		name: name,
		// we need this to be unique per process (some names are still the same) but obvious who it belongs to
		metrics:       newReflectorMetrics(makeValidPrometheusMetricLabel(fmt.Sprintf("reflector_"+name+"_%d", reflectorSuffix))),
		listerWatcher: lw,
		store:         store,
		expectedType:  reflect.TypeOf(expectedType),
		period:        time.Second, // Re-execution interval of Run() method wait.Until()
		resyncPeriod:  resyncPeriod, // Time interval for periodically calling the Reync () method of store
		clock:         &clock.RealClock{},
	}
	return r
}

Relector objects are generally created by Informer's controller, for example (specific reference: 4.controller-informer.md): 

// Source: k8s.io/client-go/tools/cache/controller.go
func (c *controller) Run(stopCh <-chan struct{}) {
	...
	// Create Reflector with controller's Config parameter
	r := NewReflector(
		c.config.ListerWatcher,
		c.config.ObjectType,
		c.config.Queue, // DeltaFIFO
		c.config.FullResyncPeriod,
	)
	r.ShouldResync = c.config.ShouldResync
	r.clock = c.clock
	...
	// Run Method for Running Reflector
	wg.StartWithChannel(stopCh, r.Run)
	...
}

Run() method

The Run() method always runs the ListAndWatch() method, and if something goes wrong, it waits for r.period time to execute the ListAndWatch() method again, so the Run() method will not return until stopCh is closed.

// From: k8s.io/client-go/tools/cache/reflector.go
func (r *Reflector) Run(stopCh <-chan struct{}) {
	klog.V(3).Infof("Starting reflector %v (%s) from %s", r.expectedType, r.resyncPeriod, r.name)
	wait.Until(func() {
		if err := r.ListAndWatch(stopCh); err != nil {
			utilruntime.HandleError(err)
		}
	}, r.period, stopCh)
}

ListAndWatch() method

This method is the core method of Refelector, which implements:

  1. All objects of the apiserver List resource type (ResourceVersion 0);
  2. Get the resourceVersion of this type of object from the list of objects;
  3. Call the Replace() method of the internal Store (DeltaFIFO) to update the List object to the internal DeltaFIFO (generate Sync events, or Deleted events of Deleted FinalState Unknown type);
  4. The resyncPeriod calls the Rescycn () method of DeltaFIFO to synchronize the objects in the knownObjects cache into DeltaFIFO (SYNC events), so as to realize the function of processing all resource type objects periodically.
  5. The resourceVersion obtained from List starts blocking Watch apiserver and updates DeltaFIFO according to the type of event received;
// From: k8s.io/client-go/tools/cache/reflector.go

// The minimum timeout time of a Watch is actually a random value between [minWatch Timeout, 2 * minWatch Timeout]
var minWatchTimeout = 5 * time.Minute

func (r *Reflector) ListAndWatch(stopCh <-chan struct{}) error {
	...
	// ResourceVersion: "0" denotes the current value of the object in etcd, and lists all objects of this type currently in etcd
	options := metav1.ListOptions{ResourceVersion: "0"}
	...
	list, err := r.listerWatcher.List(options)
	...
	listMetaInterface, err := meta.ListAccessor(list)
	...
	// Get the resourceVersion of this type of object from the list of objects, and use it in subsequent Watch es
	resourceVersion = listMetaInterface.GetResourceVersion()
	...
	items, err := meta.ExtractList(list)
	...
	// Synchronize items to r.store, or DeltaFIFO, using DeltaFIFO's Raplace () method
	if err := r.syncWith(items, resourceVersion); err != nil {
		return fmt.Errorf("%s: Unable to sync list result: %v", r.name, err)
	}
	// Caching resourceVersion
	r.setLastSyncResourceVersion(resourceVersion)
	...
	
    go func() {
		resyncCh, cleanup := r.resyncChan()
		defer func() {
			cleanup() // Call the last one written into cleanup
		}()
		// The Reync () method of r.store is called periodically to synchronize the objects in the knownObjects object cache into DeltaFIFO.
		for {
			select {
			case <-resyncCh:
			case <-stopCh:
				return
			case <-cancelCh:
				return
			}
			if r.ShouldResync == nil || r.ShouldResync() {
				klog.V(4).Infof("%s: forcing resync", r.name)
				if err := r.store.Resync(); err != nil {
					resyncerrc <- err
					return
				}
			}
			cleanup()
			resyncCh, cleanup = r.resyncChan()
		}
	}()

    for {
        ...
        // Watch will timeout after timeout Seconds, when r.watchHandler() error, ListAndWatch() method error return
        // Reflecter waits for the r.period event to re-execute the ListAndWatch() method.
        timeoutSeconds := int64(minWatchTimeout.Seconds() * (rand.Float64() + 1.0))
		options = metav1.ListOptions{
			ResourceVersion: resourceVersion,
			TimeoutSeconds: &timeoutSeconds,
		}
        w, err := r.listerWatcher.Watch(options)
		...
		// Blocking Handling Watch Events
        if err := r.watchHandler(w, &resourceVersion, resyncerrc, stopCh); err != nil {
			if err != errorStopRequested {
				klog.Warningf("%s: watch of %v ended with: %v", r.name, r.expectedType, err)
			}
			return nil
		}
    }
}

// Replace r.store with objects in items
func (r *Reflector) syncWith(items []runtime.Object, resourceVersion string) error {
	found := make([]interface{}, 0, len(items))
	for _, item := range items {
		found = append(found, item)
	}
	return r.store.Replace(found, resourceVersion)
}

// According to the type of event Watch arrives at, the method of r.store is called to update the object to r.store.
func (r *Reflector) watchHandler(w watch.Interface, resourceVersion *string, errc chan error, stopCh <-chan struct{}) error {
...
loop:
	for {
		select {
        ...
		case event, ok := <-w.ResultChan():
            ...
			newResourceVersion := meta.GetResourceVersion()
			switch event.Type {
			case watch.Added:
				err := r.store.Add(event.Object)
				if err != nil {
					utilruntime.HandleError(fmt.Errorf("%s: unable to add watch event object (%#v) to store: %v", r.name, event.Object, err))
				}
			case watch.Modified:
				err := r.store.Update(event.Object)
				if err != nil {
					utilruntime.HandleError(fmt.Errorf("%s: unable to update watch event object (%#v) to store: %v", r.name, event.Object, err))
				}
			case watch.Deleted:
				err := r.store.Delete(event.Object)
				if err != nil {
					utilruntime.HandleError(fmt.Errorf("%s: unable to delete watch event object (%#v) from store: %v", r.name, event.Object, err))
				}
			default:
				utilruntime.HandleError(fmt.Errorf("%s: unable to understand watch event %#v", r.name, event))
			}
			*resourceVersion = newResourceVersion
			r.setLastSyncResourceVersion(newResourceVersion)
			eventCount++
		}
	}
    ...
}

controller using Reflector

Controller encapsulates Reflector. Reflector uses Lister Watcher to retrieve object lists and events from apiserver and store them in Delta FIFO. Controller continuously pops up Delta FIFO of Reflector, updates its ClientState cache with the pop-up Delta, and calls OnAdd/Ondate/OnDelete callback function set by Informer.

// Source: k8s.io/client-go/tools/cache/controller.go
type Config struct {
    // The queue that caches ObjectType objects is also used by Reflector.
    // When subsequent NewInformer and NewIndexerInformer create this configuration, they actually create a Queue of DeltaFIFO type.
	Queue

    // Controller creates Lister Watcher for ObjectType Reflector;
	ListerWatcher

    // For each event of the object, the handler is called
	Process ProcessFunc

    // Object types that the Controller focuses on and manages
	ObjectType runtime.Object

	// Periodic call to Queue's Reync () method
	FullResyncPeriod time.Duration

	// External judgment of whether a function of Resync() is needed (usually nil)
	ShouldResync ShouldResyncFunc

	// If Process fails to process pop-up objects, do you add objects back to Queue (usually false)
	RetryOnError bool
}

// Controller is an implementation of Controller, but there is no New method to create it.
// So the controller is actually created and used by NewInformer and NewIndexer Informer.
type controller struct {
	config         Config
	reflector      *Reflector
	reflectorMutex sync.RWMutex
	clock          clock.Clock
}

Run() method

func (c *controller) Run(stopCh <-chan struct{}) {
	defer utilruntime.HandleCrash()
	go func() {
		<-stopCh
        // Close the queue at the end of the controller run
		c.config.Queue.Close()
	}()

    // Initialize Reflector for monitoring ObjectType objects according to Controller configuration
    // When Refector is initialized, the Queue(DeltaFIFO) of Controller is passed in, so the Refector updates the Queue synchronously.
	r := NewReflector(
		c.config.ListerWatcher,
		c.config.ObjectType,
		c.config.Queue,
		c.config.FullResyncPeriod,
	)
	r.ShouldResync = c.config.ShouldResync
	r.clock = c.clock

	c.reflectorMutex.Lock()
	c.reflector = r
	c.reflectorMutex.Unlock()

	var wg wait.Group
	defer wg.Wait()

    // Start Reflector in another goroutine
	wg.StartWithChannel(stopCh, r.Run)

	// Blocking executes c.processLoop, which is a dead Loop that returns the value only when an error occurs, and then waits for 1s to execute again.
	wait.Until(c.processLoop, time.Second, stopCh)
}

processLoop() method

The Deltas event pops up from DeltaFIFO and then calls the configured PopProcessFunc function, which:

  1. Update the knownObjests object cache (client state created by controller) used by DeltaFIFO with pop-up objects;
  2. Call the callback function registered by the user.

DelteFIFO's Pop() method executes PopProcessFunc under lock condition, so even if multiple goroutines call the Pop() method concurrently, they are executed serially, so there will not be multiple goroutines handling a resource object at the same time.

func (c *controller) processLoop() {
	for {
		obj, err := c.config.Queue.Pop(PopProcessFunc(c.config.Process))
		if err != nil {
			if err == FIFOClosedError {
				return
			}
			if c.config.RetryOnError {
				// This is the safe way to re-enqueue.
				c.config.Queue.AddIfNotPresent(obj)
			}
		}
	}
}

c.config.RetryOnError is generally false (refer to the subsequent NewInformer() function), so when PopProcessFunc performs an error, it does not add objects to DeltaFIFO.

Informer Using controller

NewInformer(), NewIndexInformer() functions use controller s to List/Watch objects of specific resource types, cache them locally, and invoke user-provided callback functions (stored in ResourceEventHandler).

// Source: k8s.io/client-go/tools/cache/controller.go
func NewInformer(
	lw ListerWatcher,
	objType runtime.Object,
	resyncPeriod time.Duration,
	h ResourceEventHandler,
) (Store, Controller) {
    // Store is a memory database that stores objects. It uses the KeyFunc function to get the unique access key of the object.
    // NewStore actually returns a ThreadSafe type
	clientState := NewStore(DeletionHandlingMetaNamespaceKeyFunc)
	fifo := NewDeltaFIFO(MetaNamespaceKeyFunc, clientState)

	cfg := &Config{
		Queue:            fifo,
		ListerWatcher:    lw,
		ObjectType:       objType,
		FullResyncPeriod: resyncPeriod,
		// Do not put the object that failed to execute the failed Process back to Queue
		RetryOnError:     false,

		// DeltaFIFO's Pop() method executes the Process function with internal locking, so updating and calling the OnUpdate/OnAdd/OnDelted processing function for clientState is serial.
		Process: func(obj interface{}) error {
			// from oldest to newest
			for _, d := range obj.(Deltas) {
				switch d.Type {
				case Sync, Added, Updated:
					// Decide whether to call OnUpdate() or OnAdd() Handler Based on whether or not the object is in clientState
					// So, when Controller starts, because clientState is empty, OnAdd() handler is called for all the objects to which the list arrives.
					if old, exists, err := clientState.Get(d.Object); err == nil && exists {
						if err := clientState.Update(d.Object); err != nil {
							return err
						}
						h.OnUpdate(old, d.Object)
					} else {
						if err := clientState.Add(d.Object); err != nil {
							return err
						}
						h.OnAdd(d.Object)
					}
				case Deleted:
					// When deleting, the object is deleted from the clientState first, and then the user processing function is called.
					if err := clientState.Delete(d.Object); err != nil {
						return err
					}
					// d.Object may be either a native resource type object or a Deleted FinalState Unknowown type object, so the OnDeleted() function needs to be able to distinguish and process it.
					h.OnDelete(d.Object)
				}
			}
			return nil
		},
	}
	return clientState, New(cfg)
}

HasSynced() method

Call the HasSynced() method of DeltaFIFO.

When processLoop() pops up the first Reflector List objects in Delta FIFO and ends processing, the method returns true, and then returns true all the time.

func (c *controller) HasSynced() bool {
	return c.config.Queue.HasSynced()
}

sharedInformer/sharedIndexInformer's HasSynced() method actually calls the controller's HasSynced() method, and the signature of the method is the same as that of the InformerSynced function type:

// Source: k8s.io/client-go/tools/cache/shared_informer.go
func (s *sharedIndexInformer) HasSynced() bool {
	s.startedLock.Lock()
	defer s.startedLock.Unlock()

	if s.controller == nil {
		return false
	}
	return s.controller.HasSynced()
}

type InformerSynced func() bool

Customize Controller scenarios using HasSynced() methods

When developing K8S Controller, a convention is to call cache.WaitForCacheSync and wait for all Informer Cache to synchronize before starting the worker of consuming workqueue:

// Source: https://github.com/kubernetes/sample-controller/blob/master/controller.go

// Customized Controller
type Controller struct {
	...
	deploymentsLister appslisters.DeploymentLister
	deploymentsSynced cache.InformerSynced  // InformerSynced function type
	...
}

func NewController(
	kubeclientset kubernetes.Interface,
	sampleclientset clientset.Interface,
	deploymentInformer appsinformers.DeploymentInformer,
	fooInformer informers.FooInformer) *Controller {
		...
		controller := &Controller{
			kubeclientset:     kubeclientset,
			sampleclientset:   sampleclientset,
			deploymentsLister: deploymentInformer.Lister(),
			deploymentsSynced: deploymentInformer.Informer().HasSynced, // Infomer's HasSynced() method
		}
		...
}

// Wait for all types of Informer's HasSynced() method to return to true before starting workers
func (c *Controller) Run(threadiness int, stopCh <-chan struct{}) error {
	...
	// Wait for the caches to be synced before starting workers
	klog.Info("Waiting for informer caches to sync")
	if ok := cache.WaitForCacheSync(stopCh, c.deploymentsSynced, c.foosSynced); !ok {
		return fmt.Errorf("failed to wait for caches to sync")
	}

	klog.Info("Starting workers")
	// Launch two workers to process Foo resources
	for i := 0; i < threadiness; i++ {
		go wait.Until(c.runWorker, time.Second, stopCh)
	}
	...
}

Why wait until the HasSynced() of informer returns true to start the worker?

Because when HasSynced() returns true, it indicates that the first batch of Reflecter List objects pop up from DeltaFIFO and are updated to the clientState cache by the controller, so that the worker can get to the object from the Lister Get through the object name (Key). Otherwise, the object may still be in DeltaFIFO and not synchronized to the clientState cache, so that the worker can not get the object from the Lister by its object name.

Topics: Kubernetes REST Nginx Database

Hot Topics

©2024 Programmer Think