Take you to understand K8s DeltaFIFO from five functions

Posted by jmantra on Thu, 24 Feb 2022 09:15:58 +0100

Abstract: DeltaFIFO is a Queue used to store and process data in K8s. Compared with traditional FIFO, it not only stores data and ensures first in first out, but also stores the type of K8s resource object. It is an important channel connecting reflector (producer) and indexer (consumer).

This article is shared from Huawei cloud community< DeltaFIFO of client go source code analysis >, by kaliarch.

Compared with 8kqueue, it is not only used to store data in 8kqueue, but also used to process data first. It is an important channel connecting reflector (producer) and indexer (consumer).

One source code

Here, we focus on several important methods and functions to understand several important attributes and methods of DeltaFIFO. For more detailed methods such as Queue and FIFO, we need to check the more detailed source code.

1.1 DeltaFIFO structure

// Delta structure
type Delta struct {
	Type   DeltaType
	Object interface{}
}

type DeltaType string

// Change type definition
const (
	Added   DeltaType = "Added"
	Updated DeltaType = "Updated"
	Deleted DeltaType = "Deleted"
 // When you encounter a watch error and have to re list, it will trigger Replaced.
  // We don't know whether the replaced object has changed.
	Replaced DeltaType = "Replaced"
	// Sync is for synthetic events during periodic resynchronization
	Sync DeltaType = "Sync"
)

type DeltaFIFO struct {
	// lock/cond protects access to "projects" and "queues" to ensure thread safety
	lock sync.RWMutex
  // cond implements a condition variable and a collection point, which is used to wait or announce the goroutine event.
  // Each Cond has a lock L (usually * Mutex or * RWMutex),
  // When you change the condition, you must keep the time when you call the Wait method. Conditions shall not be copied after the first use.
	cond sync.Cond

	// `items` maps a key to a Deltas.
	// items is the delta change list of objects of the same type
	items map[string]Deltas

	// `queue` maintains FIFO order of keys for consumption in Pop().
	// There are no duplicates in `queue`.
	// In order to ensure the order
	queue []string

  // populated is set to true if the first fill is completed by calling Replace()
	populated bool
	// Number of items inserted by the first call to Replace
	initialPopulationCount int

	// Calculate the key of item
	keyFunc KeyFunc

	// It's the indexer in the back
	knownObjects KeyListerGetter

	closed bool

	// emitDeltaTypeReplaced is whether to emit the Replaced or Sync
	// DeltaType when Replace() is called (to preserve backwards compat).
	emitDeltaTypeReplaced bool
}

From the source code, we can see that there are five types of Delta. The previous additions, deletions and modifications, as the name suggests, are used to monitor the changes of objects in the watch. Replaced and Sync are used for the first time and exceptions to ensure that the data in the indexer is consistent with that in etcd.

We can see several important attributes in DeltaFIFO.

  • queue: the key that stores the resource object.
  • items: store a kind of behavior of an object and save a series of change behaviors of the same object in sequence. key is the value calculated by keyFunc and value is the list of delta.
  • keyFunc: used to calculate the key of items.

DeltaFIFO will retain the operation type of resource object obj. There will be the same resource object with different operation types in the queue. The key of queue is calculated by Keyof function, the items field is stored by map data structure, and the value stores the deltas array of the object.

For example, if the user creates a Pod, the Delat will say a Pod with Added type. In order to follow up different operations, the controller executes different business logic.

1.2 queueActionLocked

Let's check the methods of DeltaFIFO. For example, Add/Update/Delete all call the queueActionLocked method. Analyze this method in detail.

// Add method of DeltaFIFO
func (f *DeltaFIFO) Add(obj interface{}) error {
   f.lock.Lock()
   defer f.lock.Unlock()
   f.populated = true
   return f.queueActionLocked(Added, obj)
}
  • queueActionLocked

The general steps can be divided into: obtaining the object key, adding the new object to the object list, de duplicating the delete type, storing its key in the queue if the object is not in the queue, and finally adding the object to items and updating the queue.

// 
func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error {
  //  Get object key
	id, err := f.KeyOf(obj)
	if err != nil {
		return KeyError{obj, err}
	}
  // Temporary storage of old object list
	oldDeltas := f.items[id]
  // Adds an object to the new object list
	newDeltas := append(oldDeltas, Delta{actionType, obj})
  // The delete type is de duplicated, because the update type may update a field with an exception.
	newDeltas = dedupDeltas(newDeltas)
  // Judge whether the object key is stored in the queue. If it does not exist, add it to the queue before
	if len(newDeltas) > 0 {
		if _, exists := f.items[id]; !exists {
			f.queue = append(f.queue, id)
		}
		f.items[id] = newDeltas
    // Notify all consumers to unblock
		f.cond.Broadcast()
	} else {
		// This never happens, because dedupDeltas never returns an empty list
		// when given a non-empty list (as it is here).
		// If somehow it happens anyway, deal with it but complain.
		if oldDeltas == nil {
			klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; ignoring", id, oldDeltas, obj)
			return nil
		}
		klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; breaking invariant by storing empty Deltas", id, oldDeltas, obj)
    // Generate final object items
		f.items[id] = newDeltas
		return fmt.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; broke DeltaFIFO invariant by storing empty Deltas", id, oldDeltas, obj)
	}
	return nil
}
  • KeyOf

The KeyOf method is used to obtain the latest object key of DeltaFIFO in queueActionLocked. First, judge whether the object is a delta slice. Operate with the latest delta object and directly call the keyFunc in DeltaFIFO. If it is not passed in in K8s, the MetaNamespaceKeyFunc is used by default. The function returns < namespace > / < name >, unless < namespace > is empty, The object name is directly used as the key.

// DeltaFIFO object key calculation function
func (f *DeltaFIFO) KeyOf(obj interface{}) (string, error) {
 // Determine whether it is Delta slice
  if d, ok := obj.(Deltas); ok {
  if len(d) == 0 {
   return "", KeyError{obj, ErrZeroLengthDeltasObject}
  }
  // Calculate using the latest version of the object
  obj = d.Newest().Object
 }
 if d, ok := obj.(DeletedFinalStateUnknown); ok {
  return d.Key, nil
 }
  // The specific calculation depends on initializing the KeyFunc function passed in from DeltaFIFO
 return f.keyFunc(obj)
}

// Newest returns the latest Delta, or nil if not.
func (d Deltas) Newest() *Delta {
 if n := len(d); n > 0 {
  return &d[n-1]
 }
 return nil
}

1.3 Replace

In the previous Reflector learning, we can see that in the ListAndWatch method, the last call to the full List of the resource is actually the Replace method in the Reflector incoming Store.

The replace method is mainly used for the full update of objects. Since the external output of DeltaFIFO is the incremental change of all targets, it is necessary to judge whether the object has been deleted every full update, because the request for target deletion may not be received before the full update. This is different from cache. Replace() of cache is equivalent to reconstruction, because cache is a memory mapping of the full amount of objects, so Replace() is equivalent to reconstruction.

func (f *DeltaFIFO) Replace(list []interface{}, resourceVersion string) error {
	f.lock.Lock()
	defer f.lock.Unlock()
  // Construct a set
	keys := make(sets.String, len(list))

	// The List operation of the old version client is Sync, and this is for backward compatibility
	action := Sync
	if f.emitDeltaTypeReplaced {
		action = Replaced
	}

	// Traverse the incoming object column slice and add it to DeltaFIFO.
	for _, item := range list {
    // Get object key
		key, err := f.KeyOf(item)
		if err != nil {
			return KeyError{item, err}
		}
    // Use the Set set to save the processed object keys
		keys.Insert(key)
		if err := f.queueActionLocked(action, item); err != nil {
			return fmt.Errorf("couldn't enqueue object: %v", err)
		}
	}

  // Judge whether there is Indexer storage. If there is no Indexer, maintain your own Queue
  // If the old object is not in the Queue, delete the object, otherwise update to the latest object
	if f.knownObjects == nil {
		// Do deletion detection against our own list.
		queuedDeletions := 0
 
    // Perform updates on objects
		for k, oldItem := range f.items {
			if keys.Has(k) {
				continue
			}
			// Delete pre-existing items not in the new list.
			// This could happen if watch deletion event was missed while
			// disconnected from apiserver.
			var deletedObj interface{}
			if n := oldItem.Newest(); n != nil {
				deletedObj = n.Object
			}
			queuedDeletions++
      // Because there may already be Deleted elements in the queue to avoid duplication, DeletedFinalStateUnknown is adopted 
			if err := f.queueActionLocked(Deleted, DeletedFinalStateUnknown{k, deletedObj}); err != nil {
				return err
			}
		}
   // If populated is false, it indicates that the queue object entered for the first time is the completion of the operation
		if !f.populated {
			f.populated = true
			// Record the number of objects set for the first time
			f.initialPopulationCount = keys.Len() + queuedDeletions
		}

		return nil
	}

	// Detect deleted objects that are not in the queue
	knownKeys := f.knownObjects.ListKeys()
	queuedDeletions := 0
	for _, k := range knownKeys {
		if keys.Has(k) {
			continue
		}
    // Get from indexer according to key
		deletedObj, exists, err := f.knownObjects.GetByKey(k)
		if err != nil {
			deletedObj = nil
			klog.Errorf("Unexpected error %v during lookup of key %v, placing DeleteFinalStateUnknown marker without object", err, k)
		} else if !exists {
			deletedObj = nil
			klog.Infof("Key %v does not exist in known objects store, placing DeleteFinalStateUnknown marker without object", k)
		}
		queuedDeletions++
    // Put the deleted delta of the object into the queue
		if err := f.queueActionLocked(Deleted, DeletedFinalStateUnknown{k, deletedObj}); err != nil {
			return err
		}
	}

	if !f.populated {
		f.populated = true
		f.initialPopulationCount = keys.Len() + queuedDeletions
	}

	return nil
}

1.4 Resync

In timing synchronization, the Resync method of Store is invoked, Resync is resynchronization, and the Delta object with Sync type. If f.knownObjects is Indexer does not exist, Resync operation is not performed.

func (f *DeltaFIFO) Resync() error {
	f.lock.Lock()
	defer f.lock.Unlock()
	// If there is no indexer, it will not be executed
	if f.knownObjects == nil {
		return nil
	}
  // Get the key list of indexer
	keys := f.knownObjects.ListKeys()
	for _, k := range keys {
		if err := f.syncKeyLocked(k); err != nil {
			return err
		}
	}
	return nil
}
// Specific resync operations
func (f *DeltaFIFO) syncKeyLocked(key string) error {
  // Get key
	obj, exists, err := f.knownObjects.GetByKey(key)
  // If the error or does not exist, return directly
	if err != nil {
		klog.Errorf("Unexpected error %v during lookup of key %v, unable to queue object for sync", err, key)
		return nil
	} else if !exists {
		klog.Infof("Key %v does not exist in known objects store, unable to queue object for sync", key)
		return nil
	}

	// If we are doing Resync() and there is already an event queued for that object,
	// we ignore the Resync for it. This is to avoid the race, in which the resync
	// comes with the previous value of object (since queueing an event for the object
	// doesn't trigger changing the underlying store <knownObjects>.
  // Get the key according to KeyOf
	id, err := f.KeyOf(obj)
	if err != nil {
		return KeyError{obj, err}
	}
  // If the key has a value in DeltaFIFO, it will not be updated for the time being. After the last object operation is completed, the update of the last status will be executed
	if len(f.items[id]) > 0 {
		return nil
	}
  // Add this Delta for object synchronization
	if err := f.queueActionLocked(Sync, obj); err != nil {
		return fmt.Errorf("couldn't queue object: %v", err)
	}
	return nil
}

1.5 Pop

Finally, let's look at the consumption of objects in DeltaFIFO. In fact, pop function is used. The specific processing of data flow is realized through PopProcessFunc. Pop will wait until an element is ready before processing. If multiple elements are ready, they will be returned in the order they are added or updated. Before processing, the element will be removed from the queue (and storage), so if it is not processed successfully, it should be added back with the AddIfNotPresent() function.
The processing function is called when there is a lock, so it is safe to update the data structure that needs to be synchronized with the queue.

func (f *DeltaFIFO) Pop(process PopProcessFunc) (interface{}, error) {
   f.lock.Lock()
   defer f.lock.Unlock()
   for {
      for len(f.queue) == 0 {
         // When the queue is empty, the call to Pop() will be blocked until a new element is inserted into the queue
         // When Close() is called, set f.closed and broadcast the condition.
         if f.closed {
            return nil, ErrFIFOClosed
         }

         f.cond.Wait()
      }
      // Get the first entry element for processing
      id := f.queue[0]
      // Delete the first element from the queue
      f.queue = f.queue[1:]
      if f.initialPopulationCount > 0 {
         f.initialPopulationCount--
      }
      // Get the ejected object
      item, ok := f.items[id]
      if !ok {
         // This should never happen
         klog.Errorf("Inconceivable! %q was in f.queue but not f.items; ignoring.", id)
         continue
      }
      // Delete pop-up elements from items
      delete(f.items, id)
      // Use process to process item s
      err := process(item)
      if e, ok := err.(ErrRequeue); ok {
         // If the processing is not successful, addIfNotPresent needs to be called to add back the queue
         f.addIfNotPresent(id, item)
         err = e.Err
      }
      // Don't need to copyDeltas here, because we're transferring
      // ownership to the caller.
      return item, err
   }
}

Two small ox knife

package main

import (
	"fmt"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/fields"
	"k8s.io/apimachinery/pkg/util/wait"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/cache"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
	"path/filepath"
	"time"
)

func Must(e interface{}) {
	if e != nil {
		panic(e)
	}
}

func InitClientSet() (*kubernetes.Clientset, error) {
	kubeconfig := filepath.Join(homedir.HomeDir(), ".kube", "config")
	restConfig, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
	if err != nil {
		return nil, err
	}
	return kubernetes.NewForConfig(restConfig)
}

// Generate listwatcher
func InitListerWatcher(clientSet *kubernetes.Clientset, resource, namespace string, fieldSelector fields.Selector) cache.ListerWatcher {
	restClient := clientSet.CoreV1().RESTClient()
	return cache.NewListWatchFromClient(restClient, resource, namespace, fieldSelector)
}

// Generate pods reflector
func InitPodsReflector(clientSet *kubernetes.Clientset, store cache.Store) *cache.Reflector {
	resource := "pods"
	namespace := "default"
	resyncPeriod := 0 * time.Second
	expectedType := &corev1.Pod{}
	lw := InitListerWatcher(clientSet, resource, namespace, fields.Everything())

	return cache.NewReflector(lw, expectedType, store, resyncPeriod)
}

// Generate DeltaFIFO
func InitDeltaQueue(store cache.Store) cache.Queue {
	return cache.NewDeltaFIFOWithOptions(cache.DeltaFIFOOptions{
		// store implements KeyListerGetter
		KnownObjects: store,
		// EmitDeltaTypeReplaced indicates that the queue consumer understands the Replaced DeltaType.
		// Before adding the 'Replaced' event type, the call to Replace() is handled in the same way as Sync().
		// false by default for backward compatibility purposes.
		// When true, a replace event is sent for the item passed to the Replace() call. When false, the 'Sync' event will be sent.
		EmitDeltaTypeReplaced: true,
	})

}
func InitStore() cache.Store {
	return cache.NewStore(cache.MetaNamespaceKeyFunc)
}

func main() {
	clientSet, err := InitClientSet()
	Must(err)
	// Used to get in processfunc
	store := InitStore()
	// queue
	DeleteFIFOQueue := InitDeltaQueue(store)
	// Generate podReflector
	podReflector := InitPodsReflector(clientSet, DeleteFIFOQueue)

	stopCh := make(chan struct{})
	defer close(stopCh)
	go podReflector.Run(stopCh)
	// Process a single element. The element ke is namespace/name and the value is delta list
  // delta objects are DeltaType and runtimeobject
	ProcessFunc := func(obj interface{}) error {
		// The first received event will be processed first
		for _, d := range obj.(cache.Deltas) {
			switch d.Type {
			case cache.Sync, cache.Replaced, cache.Added, cache.Updated:
				if _, exists, err := store.Get(d.Object); err == nil && exists {
					if err := store.Update(d.Object); err != nil {
						return err
					}
				} else {
					if err := store.Add(d.Object); err != nil {
						return err
					}
				}
			case cache.Deleted:
				if err := store.Delete(d.Object); err != nil {
					return err
				}
			}
			pods, ok := d.Object.(*corev1.Pod)
			if !ok {
				return fmt.Errorf("not config: %T", d.Object)
			}

			fmt.Printf("Type:%s: Name:%s\n", d.Type, pods.Name)
		}
		return nil
	}

	fmt.Println("Start syncing...")

	wait.Until(func() {
		for {
			_, err := DeleteFIFOQueue.Pop(ProcessFunc)
			Must(err)
		}
	}, time.Second, stopCh)
}

First, create a store. After creating DeltaFIFO, initialize the Reflector and insert DeltaFIFO as a store. After the Reflector runs, perform ListWatch operation on K8s APIserver, store the data of the List in DeltaFIFO, and process the elements of DeltaFIFO through custom ProcessFunc.

III. process summary

Reflector first obtains the full amount of resource object data through ListAndWatch, then calls the Replace() method of DeltaFIFO to insert the queue into the queue. If the timing synchronization is set, the contents of Indexer are updated regularly. Then, Add, Update, Delete methods of DeltaFIFO are invoked according to the operation type of the resource object through Watch operation. How to deal with Pop's elements depends on Pop's callback function PopProcessFunc.

Reference link

  • https://cloud.tencent.com/developer/article/1692474

 

Click follow to learn about Huawei's new cloud technology for the first time~

Topics: delta