This article is shared from Huawei cloud community< DeltaFIFO of client go source code analysis >, by kaliarch.
Compared with 8kqueue, it is not only used to store data in 8kqueue, but also used to process data first. It is an important channel connecting reflector (producer) and indexer (consumer).
One source code
Here, we focus on several important methods and functions to understand several important attributes and methods of DeltaFIFO. For more detailed methods such as Queue and FIFO, we need to check the more detailed source code.
1.1 DeltaFIFO structure
// Delta structure type Delta struct { Type DeltaType Object interface{} } type DeltaType string // Change type definition const ( Added DeltaType = "Added" Updated DeltaType = "Updated" Deleted DeltaType = "Deleted" // When you encounter a watch error and have to re list, it will trigger Replaced. // We don't know whether the replaced object has changed. Replaced DeltaType = "Replaced" // Sync is for synthetic events during periodic resynchronization Sync DeltaType = "Sync" ) type DeltaFIFO struct { // lock/cond protects access to "projects" and "queues" to ensure thread safety lock sync.RWMutex // cond implements a condition variable and a collection point, which is used to wait or announce the goroutine event. // Each Cond has a lock L (usually * Mutex or * RWMutex), // When you change the condition, you must keep the time when you call the Wait method. Conditions shall not be copied after the first use. cond sync.Cond // `items` maps a key to a Deltas. // items is the delta change list of objects of the same type items map[string]Deltas // `queue` maintains FIFO order of keys for consumption in Pop(). // There are no duplicates in `queue`. // In order to ensure the order queue []string // populated is set to true if the first fill is completed by calling Replace() populated bool // Number of items inserted by the first call to Replace initialPopulationCount int // Calculate the key of item keyFunc KeyFunc // It's the indexer in the back knownObjects KeyListerGetter closed bool // emitDeltaTypeReplaced is whether to emit the Replaced or Sync // DeltaType when Replace() is called (to preserve backwards compat). emitDeltaTypeReplaced bool }
From the source code, we can see that there are five types of Delta. The previous additions, deletions and modifications, as the name suggests, are used to monitor the changes of objects in the watch. Replaced and Sync are used for the first time and exceptions to ensure that the data in the indexer is consistent with that in etcd.
We can see several important attributes in DeltaFIFO.
- queue: the key that stores the resource object.
- items: store a kind of behavior of an object and save a series of change behaviors of the same object in sequence. key is the value calculated by keyFunc and value is the list of delta.
- keyFunc: used to calculate the key of items.
DeltaFIFO will retain the operation type of resource object obj. There will be the same resource object with different operation types in the queue. The key of queue is calculated by Keyof function, the items field is stored by map data structure, and the value stores the deltas array of the object.
For example, if the user creates a Pod, the Delat will say a Pod with Added type. In order to follow up different operations, the controller executes different business logic.
1.2 queueActionLocked
Let's check the methods of DeltaFIFO. For example, Add/Update/Delete all call the queueActionLocked method. Analyze this method in detail.
// Add method of DeltaFIFO func (f *DeltaFIFO) Add(obj interface{}) error { f.lock.Lock() defer f.lock.Unlock() f.populated = true return f.queueActionLocked(Added, obj) }
- queueActionLocked
The general steps can be divided into: obtaining the object key, adding the new object to the object list, de duplicating the delete type, storing its key in the queue if the object is not in the queue, and finally adding the object to items and updating the queue.
// func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error { // Get object key id, err := f.KeyOf(obj) if err != nil { return KeyError{obj, err} } // Temporary storage of old object list oldDeltas := f.items[id] // Adds an object to the new object list newDeltas := append(oldDeltas, Delta{actionType, obj}) // The delete type is de duplicated, because the update type may update a field with an exception. newDeltas = dedupDeltas(newDeltas) // Judge whether the object key is stored in the queue. If it does not exist, add it to the queue before if len(newDeltas) > 0 { if _, exists := f.items[id]; !exists { f.queue = append(f.queue, id) } f.items[id] = newDeltas // Notify all consumers to unblock f.cond.Broadcast() } else { // This never happens, because dedupDeltas never returns an empty list // when given a non-empty list (as it is here). // If somehow it happens anyway, deal with it but complain. if oldDeltas == nil { klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; ignoring", id, oldDeltas, obj) return nil } klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; breaking invariant by storing empty Deltas", id, oldDeltas, obj) // Generate final object items f.items[id] = newDeltas return fmt.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; broke DeltaFIFO invariant by storing empty Deltas", id, oldDeltas, obj) } return nil }
- KeyOf
The KeyOf method is used to obtain the latest object key of DeltaFIFO in queueActionLocked. First, judge whether the object is a delta slice. Operate with the latest delta object and directly call the keyFunc in DeltaFIFO. If it is not passed in in K8s, the MetaNamespaceKeyFunc is used by default. The function returns < namespace > / < name >, unless < namespace > is empty, The object name is directly used as the key.
// DeltaFIFO object key calculation function func (f *DeltaFIFO) KeyOf(obj interface{}) (string, error) { // Determine whether it is Delta slice if d, ok := obj.(Deltas); ok { if len(d) == 0 { return "", KeyError{obj, ErrZeroLengthDeltasObject} } // Calculate using the latest version of the object obj = d.Newest().Object } if d, ok := obj.(DeletedFinalStateUnknown); ok { return d.Key, nil } // The specific calculation depends on initializing the KeyFunc function passed in from DeltaFIFO return f.keyFunc(obj) } // Newest returns the latest Delta, or nil if not. func (d Deltas) Newest() *Delta { if n := len(d); n > 0 { return &d[n-1] } return nil }
1.3 Replace
In the previous Reflector learning, we can see that in the ListAndWatch method, the last call to the full List of the resource is actually the Replace method in the Reflector incoming Store.
The replace method is mainly used for the full update of objects. Since the external output of DeltaFIFO is the incremental change of all targets, it is necessary to judge whether the object has been deleted every full update, because the request for target deletion may not be received before the full update. This is different from cache. Replace() of cache is equivalent to reconstruction, because cache is a memory mapping of the full amount of objects, so Replace() is equivalent to reconstruction.
func (f *DeltaFIFO) Replace(list []interface{}, resourceVersion string) error { f.lock.Lock() defer f.lock.Unlock() // Construct a set keys := make(sets.String, len(list)) // The List operation of the old version client is Sync, and this is for backward compatibility action := Sync if f.emitDeltaTypeReplaced { action = Replaced } // Traverse the incoming object column slice and add it to DeltaFIFO. for _, item := range list { // Get object key key, err := f.KeyOf(item) if err != nil { return KeyError{item, err} } // Use the Set set to save the processed object keys keys.Insert(key) if err := f.queueActionLocked(action, item); err != nil { return fmt.Errorf("couldn't enqueue object: %v", err) } } // Judge whether there is Indexer storage. If there is no Indexer, maintain your own Queue // If the old object is not in the Queue, delete the object, otherwise update to the latest object if f.knownObjects == nil { // Do deletion detection against our own list. queuedDeletions := 0 // Perform updates on objects for k, oldItem := range f.items { if keys.Has(k) { continue } // Delete pre-existing items not in the new list. // This could happen if watch deletion event was missed while // disconnected from apiserver. var deletedObj interface{} if n := oldItem.Newest(); n != nil { deletedObj = n.Object } queuedDeletions++ // Because there may already be Deleted elements in the queue to avoid duplication, DeletedFinalStateUnknown is adopted if err := f.queueActionLocked(Deleted, DeletedFinalStateUnknown{k, deletedObj}); err != nil { return err } } // If populated is false, it indicates that the queue object entered for the first time is the completion of the operation if !f.populated { f.populated = true // Record the number of objects set for the first time f.initialPopulationCount = keys.Len() + queuedDeletions } return nil } // Detect deleted objects that are not in the queue knownKeys := f.knownObjects.ListKeys() queuedDeletions := 0 for _, k := range knownKeys { if keys.Has(k) { continue } // Get from indexer according to key deletedObj, exists, err := f.knownObjects.GetByKey(k) if err != nil { deletedObj = nil klog.Errorf("Unexpected error %v during lookup of key %v, placing DeleteFinalStateUnknown marker without object", err, k) } else if !exists { deletedObj = nil klog.Infof("Key %v does not exist in known objects store, placing DeleteFinalStateUnknown marker without object", k) } queuedDeletions++ // Put the deleted delta of the object into the queue if err := f.queueActionLocked(Deleted, DeletedFinalStateUnknown{k, deletedObj}); err != nil { return err } } if !f.populated { f.populated = true f.initialPopulationCount = keys.Len() + queuedDeletions } return nil }
1.4 Resync
In timing synchronization, the Resync method of Store is invoked, Resync is resynchronization, and the Delta object with Sync type. If f.knownObjects is Indexer does not exist, Resync operation is not performed.
func (f *DeltaFIFO) Resync() error { f.lock.Lock() defer f.lock.Unlock() // If there is no indexer, it will not be executed if f.knownObjects == nil { return nil } // Get the key list of indexer keys := f.knownObjects.ListKeys() for _, k := range keys { if err := f.syncKeyLocked(k); err != nil { return err } } return nil } // Specific resync operations func (f *DeltaFIFO) syncKeyLocked(key string) error { // Get key obj, exists, err := f.knownObjects.GetByKey(key) // If the error or does not exist, return directly if err != nil { klog.Errorf("Unexpected error %v during lookup of key %v, unable to queue object for sync", err, key) return nil } else if !exists { klog.Infof("Key %v does not exist in known objects store, unable to queue object for sync", key) return nil } // If we are doing Resync() and there is already an event queued for that object, // we ignore the Resync for it. This is to avoid the race, in which the resync // comes with the previous value of object (since queueing an event for the object // doesn't trigger changing the underlying store <knownObjects>. // Get the key according to KeyOf id, err := f.KeyOf(obj) if err != nil { return KeyError{obj, err} } // If the key has a value in DeltaFIFO, it will not be updated for the time being. After the last object operation is completed, the update of the last status will be executed if len(f.items[id]) > 0 { return nil } // Add this Delta for object synchronization if err := f.queueActionLocked(Sync, obj); err != nil { return fmt.Errorf("couldn't queue object: %v", err) } return nil }
1.5 Pop
Finally, let's look at the consumption of objects in DeltaFIFO. In fact, pop function is used. The specific processing of data flow is realized through PopProcessFunc. Pop will wait until an element is ready before processing. If multiple elements are ready, they will be returned in the order they are added or updated. Before processing, the element will be removed from the queue (and storage), so if it is not processed successfully, it should be added back with the AddIfNotPresent() function.
The processing function is called when there is a lock, so it is safe to update the data structure that needs to be synchronized with the queue.
func (f *DeltaFIFO) Pop(process PopProcessFunc) (interface{}, error) { f.lock.Lock() defer f.lock.Unlock() for { for len(f.queue) == 0 { // When the queue is empty, the call to Pop() will be blocked until a new element is inserted into the queue // When Close() is called, set f.closed and broadcast the condition. if f.closed { return nil, ErrFIFOClosed } f.cond.Wait() } // Get the first entry element for processing id := f.queue[0] // Delete the first element from the queue f.queue = f.queue[1:] if f.initialPopulationCount > 0 { f.initialPopulationCount-- } // Get the ejected object item, ok := f.items[id] if !ok { // This should never happen klog.Errorf("Inconceivable! %q was in f.queue but not f.items; ignoring.", id) continue } // Delete pop-up elements from items delete(f.items, id) // Use process to process item s err := process(item) if e, ok := err.(ErrRequeue); ok { // If the processing is not successful, addIfNotPresent needs to be called to add back the queue f.addIfNotPresent(id, item) err = e.Err } // Don't need to copyDeltas here, because we're transferring // ownership to the caller. return item, err } }
Two small ox knife
package main import ( "fmt" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/clientcmd" "k8s.io/client-go/util/homedir" "path/filepath" "time" ) func Must(e interface{}) { if e != nil { panic(e) } } func InitClientSet() (*kubernetes.Clientset, error) { kubeconfig := filepath.Join(homedir.HomeDir(), ".kube", "config") restConfig, err := clientcmd.BuildConfigFromFlags("", kubeconfig) if err != nil { return nil, err } return kubernetes.NewForConfig(restConfig) } // Generate listwatcher func InitListerWatcher(clientSet *kubernetes.Clientset, resource, namespace string, fieldSelector fields.Selector) cache.ListerWatcher { restClient := clientSet.CoreV1().RESTClient() return cache.NewListWatchFromClient(restClient, resource, namespace, fieldSelector) } // Generate pods reflector func InitPodsReflector(clientSet *kubernetes.Clientset, store cache.Store) *cache.Reflector { resource := "pods" namespace := "default" resyncPeriod := 0 * time.Second expectedType := &corev1.Pod{} lw := InitListerWatcher(clientSet, resource, namespace, fields.Everything()) return cache.NewReflector(lw, expectedType, store, resyncPeriod) } // Generate DeltaFIFO func InitDeltaQueue(store cache.Store) cache.Queue { return cache.NewDeltaFIFOWithOptions(cache.DeltaFIFOOptions{ // store implements KeyListerGetter KnownObjects: store, // EmitDeltaTypeReplaced indicates that the queue consumer understands the Replaced DeltaType. // Before adding the 'Replaced' event type, the call to Replace() is handled in the same way as Sync(). // false by default for backward compatibility purposes. // When true, a replace event is sent for the item passed to the Replace() call. When false, the 'Sync' event will be sent. EmitDeltaTypeReplaced: true, }) } func InitStore() cache.Store { return cache.NewStore(cache.MetaNamespaceKeyFunc) } func main() { clientSet, err := InitClientSet() Must(err) // Used to get in processfunc store := InitStore() // queue DeleteFIFOQueue := InitDeltaQueue(store) // Generate podReflector podReflector := InitPodsReflector(clientSet, DeleteFIFOQueue) stopCh := make(chan struct{}) defer close(stopCh) go podReflector.Run(stopCh) // Process a single element. The element ke is namespace/name and the value is delta list // delta objects are DeltaType and runtimeobject ProcessFunc := func(obj interface{}) error { // The first received event will be processed first for _, d := range obj.(cache.Deltas) { switch d.Type { case cache.Sync, cache.Replaced, cache.Added, cache.Updated: if _, exists, err := store.Get(d.Object); err == nil && exists { if err := store.Update(d.Object); err != nil { return err } } else { if err := store.Add(d.Object); err != nil { return err } } case cache.Deleted: if err := store.Delete(d.Object); err != nil { return err } } pods, ok := d.Object.(*corev1.Pod) if !ok { return fmt.Errorf("not config: %T", d.Object) } fmt.Printf("Type:%s: Name:%s\n", d.Type, pods.Name) } return nil } fmt.Println("Start syncing...") wait.Until(func() { for { _, err := DeleteFIFOQueue.Pop(ProcessFunc) Must(err) } }, time.Second, stopCh) }
First, create a store. After creating DeltaFIFO, initialize the Reflector and insert DeltaFIFO as a store. After the Reflector runs, perform ListWatch operation on K8s APIserver, store the data of the List in DeltaFIFO, and process the elements of DeltaFIFO through custom ProcessFunc.
III. process summary
Reflector first obtains the full amount of resource object data through ListAndWatch, then calls the Replace() method of DeltaFIFO to insert the queue into the queue. If the timing synchronization is set, the contents of Indexer are updated regularly. Then, Add, Update, Delete methods of DeltaFIFO are invoked according to the operation type of the resource object through Watch operation. How to deal with Pop's elements depends on Pop's callback function PopProcessFunc.
Reference link
- https://cloud.tencent.com/developer/article/1692474
Click follow to learn about Huawei's new cloud technology for the first time~