Deeply Understanding the Principle of Go-defer

Posted by trrobnett on Fri, 06 Sep 2019 06:28:18 +0200

Defer is also a special key word in Go, which is mainly used to ensure that in the process of program execution, the function behind defer will be executed, generally used to close the connection, clean up resources and so on.

1. Overview of structure

1.1. defer

type _defer struct {
   siz     int32   // Size of parameters
   started bool    // Has it been implemented?
   sp      uintptr // sp at time of defer
   pc      uintptr
   fn      *funcval 
   _panic  *_panic // panic in defer
   link    *_defer // Defer list, defer in the function execution process, is concatenated by link
}

1.2. panic

type _panic struct {
   argp      unsafe.Pointer // pointer to arguments of deferred call run during panic; cannot move - known to liblink
   arg       interface{}    // argument to panic
   link      *_panic        // link to earlier panic
   recovered bool           // whether this panic is over
   aborted   bool           // the panic was aborted
}

1.3. g

Because defer panic is bound to running g, here are the attributes associated with defer panic in G

type g struct {
   _panic         *_panic // Link List Composed of panic
   _defer         *_defer // defer consists of a list of first-in-last-out links, on the same stack
}

2. Source code analysis

2.1. main

At first, let's go tool to analyze what functions are used to implement the bottom level.

func main() {
    defer func() {
        recover()
    }()
    panic("error")
}

go build -gcflags=all="-N -l" main.go

go tool objdump -s "main.main" main

▶ go tool objdump -s "main\.main" main | grep CALL
  main.go:4             0x4548d0                e81b00fdff              CALL runtime.deferproc(SB)              
  main.go:7             0x4548f2                e8b90cfdff              CALL runtime.gopanic(SB)                
  main.go:4             0x4548fa                e88108fdff              CALL runtime.deferreturn(SB)            
  main.go:3             0x454909                e85282ffff              CALL runtime.morestack_noctxt(SB)       
  main.go:5             0x4549a6                e8d511fdff              CALL runtime.gorecover(SB)              
  main.go:4             0x4549b5                e8a681ffff              CALL runtime.morestack_noctxt(SB)

From the decompilation results, we can see that the defer keyword first calls runtime.deferproc to define a delayed invocation object, and then calls runtime.deferreturn to complete the invocation of defer-defined functions before the end of the function.

The panic function calls runtime.gopanic to implement the relevant logic

Recovery calls runtime. gorecovery to implement recovery

2.2. deferproc

According to the function fn defined after the defer keyword and the size of the parameter, a delayed execution function is created, and the delayed function is hung on the linked list of the current g_defer.

func deferproc(siz int32, fn *funcval) { // arguments of fn follow fn
   sp := getcallersp()
   argp := uintptr(unsafe.Pointer(&fn)) + unsafe.Sizeof(fn)
   callerpc := getcallerpc()
   // Get a _defer object and put it in the head of the g._defer list
   d := newdefer(siz)
     // Set defer's fn pc sp, etc., and call later
   d.fn = fn
   d.pc = callerpc
   d.sp = sp
   switch siz {
   case 0:
      // Do nothing.
   case sys.PtrSize:
      // _ Memory behind defer stores the address information of argp
      *(*uintptr)(deferArgs(d)) = *(*uintptr)(unsafe.Pointer(argp))
   default:
      // If it is not a pointer type parameter, copy the parameter to the memory space behind _defer
      memmove(deferArgs(d), unsafe.Pointer(argp), uintptr(siz))
   }
   return0()
}

This function looks fairly straightforward. Get an object of _defer through newproc and add it to the head of the current g's _defer list. Then copy the pointer of the parameter or parameter to the memory space behind the acquired _defer object.

2.2.1. newdefer

The function of newdefer is to get a _defer object and push it into the head of the g._defer list.

func newdefer(siz int32) *_defer {
   var d *_defer
   // Judging the sizeclass to be allocated by deferclass is similar to determining several sizeclasses in advance according to memory allocation, and then determining sizeclass according to size to find the corresponding cached memory block.
   sc := deferclass(uintptr(siz))
   gp := getg()
   // If the sizeclass is within the given sizeclass range, go to p bound by g.
   if sc < uintptr(len(p{}.deferpool)) {
      pp := gp.m.p.ptr()
      if len(pp.deferpool[sc]) == 0 && sched.deferpool[sc] != nil {
         // The number of caches in the current sizeclass = 0, not nil, gets a batch of caches from scheme
         systemstack(func() {
            lock(&sched.deferlock)
            for len(pp.deferpool[sc]) < cap(pp.deferpool[sc])/2 && sched.deferpool[sc] != nil {
               d := sched.deferpool[sc]
               sched.deferpool[sc] = d.link
               d.link = nil
               pp.deferpool[sc] = append(pp.deferpool[sc], d)
            }
            unlock(&sched.deferlock)
         })
      }
      // If the cache corresponding to sizeclass is not empty after being retrieved from scheme, allocation
      if n := len(pp.deferpool[sc]); n > 0 {
         d = pp.deferpool[sc][n-1]
         pp.deferpool[sc][n-1] = nil
         pp.deferpool[sc] = pp.deferpool[sc][:n-1]
      }
   }
   // p and sched do not find or have no corresponding sizeclass, allocate directly
   if d == nil {
      // Allocate new defer+args.
      systemstack(func() {
         total := roundupsize(totaldefersize(uintptr(siz)))
         d = (*_defer)(mallocgc(total, deferType, true))
      })
   }
   d.siz = siz
   // Insert into the list header of g._defer
   d.link = gp._defer
   gp._defer = d
   return d
}

The idea of memory allocation is to acquire sizeclass according to size and classify and cache sizeclass according to size.

First allocate on p, then get local caches in batches from global scheme. This idea of secondary caching is really pervasive in all parts of go source code.

2.3. deferreturn

func deferreturn(arg0 uintptr) {
   gp := getg()
   // Get the first defer of the g defer list and the last declared defer
   d := gp._defer
   // Without defer, you don't need to do anything.
   if d == nil {
      return
   }
   sp := getcallersp()
   // If the sp of defer does not match callersp, it indicates that defer does not correspond. It may be a delay function that calls other stack frames.
   if d.sp != sp {
      return
   }
   // According to d.siz, the original stored parameter information is acquired and stored in arg0
   switch d.siz {
   case 0:
      // Do nothing.
   case sys.PtrSize:
      *(*uintptr)(unsafe.Pointer(&arg0)) = *(*uintptr)(deferArgs(d))
   default:
      memmove(unsafe.Pointer(&arg0), deferArgs(d), uintptr(d.siz))
   }
   fn := d.fn
   d.fn = nil
   // defer is released when it's used.
   gp._defer = d.link
   freedefer(d)
   // Jump to execution defer
   jmpdefer(fn, uintptr(unsafe.Pointer(&arg0)))
}

2.3.1.freedefer

The idea of releasing defer functions should be the same as that of scheduler and memory allocation.

func freedefer(d *_defer) {
   // sizeclass for judging defer
   sc := deferclass(uintptr(d.siz))
   // Beyond the established sizeclass, it's directly allocated memory, and that's all.
   if sc >= uintptr(len(p{}.deferpool)) {
      return
   }
   pp := getg().m.p.ptr()
   // The buffer corresponding to p local sizeclass is full and half of the batch transfer to global sched
   if len(pp.deferpool[sc]) == cap(pp.deferpool[sc]) {
      // Use g0 to transfer
      systemstack(func() {
         var first, last *_defer
         for len(pp.deferpool[sc]) > cap(pp.deferpool[sc])/2 {
            n := len(pp.deferpool[sc])
            d := pp.deferpool[sc][n-1]
            pp.deferpool[sc][n-1] = nil
            pp.deferpool[sc] = pp.deferpool[sc][:n-1]
            // First string the defer objects that need to be transferred into a linked list
            if first == nil {
               first = d
            } else {
               last.link = d
            }
            last = d
         }
         lock(&sched.deferlock)
         // Put this list in the header of sched.deferpool corresponding to sizeclass
         last.link = sched.deferpool[sc]
         sched.deferpool[sc] = first
         unlock(&sched.deferlock)
      })
   }
   // Clear the attributes of defer currently being released
   d.siz = 0
   d.started = false
   d.sp = 0
   d.pc = 0
   d.link = nil

   pp.deferpool[sc] = append(pp.deferpool[sc], d)
}

The idea of secondary caching is Deeply Understanding the Implementation of Go-goroutine and Scheduler AnalysisDeeply Understanding the Principles of go-channel and selectDeep Understanding of Go-Garbage Recycling Mechanism Having analyzed it, we will not analyze it too much.

2.4. gopanic

func gopanic(e interface{}) {
   gp := getg()

   var p _panic
   p.arg = e
   p.link = gp._panic
   gp._panic = (*_panic)(noescape(unsafe.Pointer(&p)))

   atomic.Xadd(&runningPanicDefers, 1)
   // Defer objects that execute the g._defer linked list in turn
   for {
      d := gp._defer
      if d == nil {
         break
      }

      // If defer was started by earlier panic or Goexit (and, since we're back here, that triggered a new panic),
      // take defer off list. The earlier panic or Goexit will not continue running.
      // Normally, defer is removed after execution. Since the defer is not removed, there are only two reasons: 1. panic 2 is triggered in this defer. runtime.Goexit is triggered in this defer, but this defer has been executed and needs to be removed. If the defer is not removed, it is the first one. The reason is that the panic also needs to be removed, because the panic has been executed. Here, add a flag to the panic for subsequent removal.
      if d.started {
         if d._panic != nil {
            d._panic.aborted = true
         }
         d._panic = nil
         d.fn = nil
         gp._defer = d.link
         freedefer(d)
         continue
      }
      d.started = true

      // Record the panic that is running the defer.
      // If there is a new panic during the deferred call, that panic
      // will find d in the list and will mark d._panic (this panic) aborted.
      // Bind the current panic to the defer, and there may be panic in the defer. In this case, it will go into the logic of d.started above, and then terminate the current panic because it has already been executed. 
      d._panic = (*_panic)(noescape(unsafe.Pointer(&p)))
      // Execute defer.fn
      p.argp = unsafe.Pointer(getargp(0))
      reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))
      p.argp = nil

      // reflectcall did not panic. Remove d.
      if gp._defer != d {
         throw("bad defer entry in panic")
      }
      // Resolve the binding relationship between defer and panic, because the defer function has been executed, if there is panic or Goexit, it will not be executed here.
      d._panic = nil
      d.fn = nil
      gp._defer = d.link

      // trigger shrinkage to test stack copy. See stack_test.go:TestStackPanic
      //GC()

      pc := d.pc
      sp := unsafe.Pointer(d.sp) // must be pointer so it gets adjusted during stack copy
      freedefer(d)
      // When panic is recover ed, there is no need to continue panic and continue executing the remaining code
      if p.recovered {
         atomic.Xadd(&runningPanicDefers, -1)

         gp._panic = p.link
         // Aborted panics are marked but remain on the g.panic list.
         // Remove them from the list.
         // Remove aborted panic from the panic list, as explained below
         for gp._panic != nil && gp._panic.aborted {
            gp._panic = gp._panic.link
         }
         if gp._panic == nil { // must be done with signal
            gp.sig = 0
         }
         // Pass information about recovering frame to recovery.
         gp.sigcode0 = uintptr(sp)
         gp.sigcode1 = pc
         // Call recovery to restore the scheduled execution of the current g
         mcall(recovery)
         throw("recovery failed") // mcall should not return
      }
   }
     // Print panic information
   preprintpanics(gp._panic)
     // panic
   fatalpanic(gp._panic) // should not return
   *(*int)(nil) = 0      // not reached
}

Here's how gp._panic.aborted works. Here's an example

func main() {
   defer func() { // defer1
      recover()
   }()
   panic1()
}

func panic1() {
   defer func() {  // defer2
      panic("error1") // panic2
   }()
   panic("error")  // panic1
}
  1. When executed to panic("error")

    g._defer list: g._defer - > defer2 - > defer1

    g._panic list: g._panic - > Panic1

  2. When executed to panic("error1")

    g._defer list: g._defer - > defer2 - > defer1

    g._panic list: g._panic - > panic 2 - > panic 1

  3. Continue to the defer1 function and recover()

    At this point, panic caused by panic2 will be restored, panic2.recovered = true. We should continue to process the next panic along the g._panic list, but we can find that panic1 has been executed, which is the logic of the following code, to remove the panic that has been executed.

    for gp._panic != nil && gp._panic.aborted {
       gp._panic = gp._panic.link
    }

The logic of panic can be sorted out.

When the program encounters panic, it will no longer continue to execute. First, it mounts the current panic on the g._panic list, starts traversing the current g._defer list, then executes the functions defined by the _defer object, etc. If the panic happens again in the process of invoking the defer function, then it executes the gopanic function. Finally, it executes the gopanic function. The loop prints all panic information and exits the current G. However, if recovery is encountered in the process of invoking defer, the schedule (mcall(recovery) continues.

2.4.1. recovery

Restore a panic g, re-enter and continue scheduling

func recovery(gp *g) {
   // Info about defer passed in G struct.
   sp := gp.sigcode0
   pc := gp.sigcode1
   // Make the deferproc for this d return again,
   // this time returning 1.  The calling function will
   // jump to the standard return epilogue.
   // Record the sp pc returned by defer
   gp.sched.sp = sp
   gp.sched.pc = pc
   gp.sched.lr = 0
   gp.sched.ret = 1
   // Restore execution scheduling
   gogo(&gp.sched)
}

2.5. gorecover

gorecovery just sets the flag bit of g._panic.recovered

func gorecover(argp uintptr) interface{} {
   gp := getg()
   p := gp._panic
   // You need to determine whether it is called in defer function based on the address of argp
   if p != nil && !p.recovered && argp == uintptr(p.argp) {
      // Set a flag, which will be judged in the gopanic above
      p.recovered = true
      return p.arg
   }
   return nil
}

2.6. goexit

We also overlooked the point that when we manually call runtime.Goexit() to exit, the defer function will also execute. Let's analyze this situation.

func Goexit() {
    // Run all deferred functions for the current goroutine.
    // This code is similar to gopanic, see that implementation
    // for detailed comments.
    gp := getg()
  // Traversing defer list
    for {
        d := gp._defer
        if d == nil {
            break
        }
    // If defer has been executed, panic bound to defer terminates
        if d.started {
            if d._panic != nil {
                d._panic.aborted = true
                d._panic = nil
            }
            d.fn = nil
      // Remove from defer list
            gp._defer = d.link
      // Release defer
            freedefer(d)
            continue
        }
    // Calling defer internal functions
        d.started = true
        reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))
        if gp._defer != d {
            throw("bad defer entry in Goexit")
        }
        d._panic = nil
        d.fn = nil
        gp._defer = d.link
        freedefer(d)
        // Note: we ignore recovers here because Goexit isn't a panic
    }
  // Call goexit0 to clear the attributes of the current g and re-enter the schedule
    goexit1()
}

2.7. Graphic analysis

Source code is not very difficult to read, if there are any doubts, I hope the following picture can solve your doubts.

Excuse me for the slight inferiority of the drawing.

Step analysis:

  1. L3: Generate a defer1 and put it on the g._defer list
  2. L11: Generate a defer2 and mount it on the g._defer list
  3. L14: panic1 calls gopanic to place the current panic on the g._panic list
  4. L14: Because panic1, extract from the head of the g._defer list to defer2 and start executing
  5. L12: Execute defer2, another panic, mounted on the g._panic list
  6. L12: Because panic2, extracting defer2 from the head of the g._defer list, it is found that defer2 has executed the removal list, and defer2 is triggered by panic1, skipping defer2 and abort panic1.
  7. L12: Continue to extract the next part of the g._defer list and extract it to defer1
  8. L5: defer1 performs recovery, recovers panic2, removes the list, and judges that the next panic, panic1, has been removed by defer2 aborted, removes Panic1
  9. Defer1 is executed, defer1 is removed

3. Associated Documents

4. Reference Documents

  • Go Language Learning Notes--Rain Traces

Topics: Go