1, Design principle

The timer of Go language has experienced many version iterations

1. Before go version 1.9, globally unique quadheap maintenance was used
2. Go 1.10-1.13 uses 64 Quad heaps globally, and each processor (P) corresponds to one Quad heap
3. After Go 1.14, each processor P directly manages a quad heap, which is triggered by the network poller

1. Global Quad heap

All timers are stored in the following structure

var timers struct {
	lock         mutex
	gp           *g
	created      bool
	sleeping     bool
	rescheduling bool
	sleepUntil   int64
	waitnote     note
	t            []*timer
}

• t in this structure is the smallest Quad heap, in which all timers will be added when running.
• The timer wakes up when the following events occur
  • Timer expired in quad heap
  • A new timer with an earlier trigger time is added to the quad heap
• Disadvantages: Global mutex has a great impact on performance

2. Split Quad stack

The global Quad heap is divided into 64 small Quad heaps.

const timersLen = 64

var timers [timersLen]struct {
	timersBucket
}

type timersBucket struct {
	lock         mutex
	gp           *g
	created      bool
	sleeping     bool
	rescheduling bool
	sleepUntil   int64
	waitnote     note
	t            []*timer
}

• If the machine processor P exceeds 64, the timers of multiple processors will be in the same bucket. Each bucket has a co process
• Disadvantages: fragmentation reduces the granularity of locks, but frequent context switching between processors and threads affects performance

3. Network poller

The quad heap is stored directly at runtime P medium

type p struct {
	...
	timersLock mutex
	timers []*timer

	numTimers     uint32
	adjustTimers  uint32
	deletedTimers uint32
	...
}

• Timers stores the minimum Quad heap of timers
• At present, the timer is triggered by the network poller and scheduler of the processor
• Advantages: it can make full use of locality and reduce context switching overhead

2, Data structure

The internal representation of the Go language timer is runtime timer

type timer struct {
	pp puintptr

	when     int64
	period   int64
	f        func(interface{}, uintptr)
	arg      interface{}
	seq      uintptr
	nextwhen int64
	status   uint32
}

• When: the time when the current timer is awakened
• period: the interval between two wakes
• f: Function called whenever the timer wakes up
• arg: parameter passed in when executing f
• nextWhen: used to set the when field when the timer is in timeModifiedXX state
• Status: status of the timer

The timer structure exposed by Go language is time Timer

type Timer struct {
	C <-chan Time
	r runtimeTimer
}

3, State machine

The runtime uses the state machine approach to handle all timers

state	explain
timerNoStatus	The status has not been set
timerWaiting	Wait for trigger
timerRuning	Run timer function
timerDelete	Deleted
timerRemoving	Being deleted
timerRemoved	Has been stopped and removed from the heap
timerModifying	Being modified
timerModifiedEarlier	Was modified to an earlier time
timeerModifiedLater	It was modified to a later time
timerMoving	Has been modified and is being moved

1. Add timer

Adding a timer will call runtime Addtimer function

func addtimer(t *timer) {
	if t.status != timerNoStatus {
		badTimer()
	}
	t.status = timerWaiting
	cleantimers(pp)
	doaddtimer(pp, t)
	wakeNetPoller(when)
}

• Status: timernostatus - > timerwaiting
• Clean up the timer in the processor
• Adds the current timer to the timer quad of the processor
• Wake up the dormant thread in the network poller

Each time a new timer is added, the blocking timer will be interrupted and the scheduler will be triggered to check whether any timer expires

2. Delete timer

You may encounter deleting timers from other processors. Delete simply marks the timer state as deleted, and then the processor of the persistent timer completes the deletion

3. Modify timer

Modifying the timer calls runtime Modtimer function

func modtimer(t *timer, when, period int64, f func(interface{}, uintptr), arg interface{}, seq uintptr) bool {
	status := uint32(timerNoStatus)
	wasRemoved := false
loop:
	for {
		switch status = atomic.Load(&t.status); status {
			...
		}
	}

	t.period = period
	t.f = f
	t.arg = arg
	t.seq = seq

	if wasRemoved {
		t.when = when
		doaddtimer(pp, t)
		wakeNetPoller(when)
	} else {
		t.nextwhen = when
		newStatus := uint32(timerModifiedLater)
		if when < t.when {
			newStatus = timerModifiedEarlier
		}
		...
		if newStatus == timerModifiedEarlier {
			wakeNetPoller(when)
		}
	}
}

• If the modified timer has been deleted, runtime is called Doaddtimer creates a new timer
• If the modified time is greater than or equal to the modified time, set the status of the timer to timerModifyLater
• If the modified time is less than the time before modification, set the timer status to timermodifyearly and trigger the scheduler to reschedule

4. Clear timer

runtime. The cleantimers function cleans the timer in the processor queue header based on the status

func cleantimers(pp *p) bool {
	for {
		if len(pp.timers) == 0 {
			return true
		}
		t := pp.timers[0]
		switch s := atomic.Load(&t.status); s {
		case timerDeleted:
			atomic.Cas(&t.status, s, timerRemoving)
			dodeltimer0(pp)
			atomic.Cas(&t.status, timerRemoving, timerRemoved)
		case timerModifiedEarlier, timerModifiedLater:
			atomic.Cas(&t.status, s, timerMoving)

			t.when = t.nextwhen

			dodeltimer0(pp)
			doaddtimer(pp, t)
			atomic.Cas(&t.status, timerMoving, timerWaiting)
		default:
			return true
		}
	}
}

• If the timer status is timerDeleted
  • Change the status of the timer to timerRemoving
  • Delete timer on top of quad heap
  • Change the timer status to timerRemoved
• If the status of the timer is timerModifiedEarlier, timerModifiedLater
  • Change the timer status to timerMoving
  • Use the next trigger time of the timer nextWhen to override when
  • Delete timer on top of quad heap
  • Add timer to Quad heap
  • Change the status of the timer to timerWaiting

5. Adjust the timer

Similar to clearing the timer, the timer in the heap will be deleted and the status will be modified to timerModifiedEarlier and timerModifiedLater. The difference is that tuning the timer traverses all the timers in the processor heap

func adjusttimers(pp *p, now int64) {
	var moved []*timer
loop:
	for i := 0; i < len(pp.timers); i++ {
		t := pp.timers[i]
		switch s := atomic.Load(&t.status); s {
		case timerDeleted:
			// Delete timer from heap
		case timerModifiedEarlier, timerModifiedLater:
			// Modify the time of the timer
		case ...
		}
	}
	if len(moved) > 0 {
		addAdjustedTimers(pp, moved)
	}
}

6. Run timer

runtime. The runTimer checks the timer at the top of the processor's Quad heap. This function also handles the deletion and update of the timer

func runtimer(pp *p, now int64) int64 {
	for {
		t := pp.timers[0]
		switch s := atomic.Load(&t.status); s {
		case timerWaiting:
			if t.when > now {
				return t.when
			}
			atomic.Cas(&t.status, s, timerRunning)
			runOneTimer(pp, t, now)
			return 0
		case timerDeleted:
			// Delete timer
		case timerModifiedEarlier, timerModifiedLater:
			// Modify the time of the timer
		case ...
		}
	}
}

If the timer at the top of the processor's Quad heap does not arrive, the trigger event will be returned directly. Otherwise, call runtime Runonetimer runs a timer at the top of the heap

func runOneTimer(pp *p, t *timer, now int64) {
	f := t.f
	arg := t.arg
	seq := t.seq

	if t.period > 0 {
		delta := t.when - now
		t.when += t.period * (1 + -delta/t.period)
		siftdownTimer(pp.timers, 0)
		atomic.Cas(&t.status, timerRunning, timerWaiting)
		updateTimer0When(pp)
	} else {
		dodeltimer0(pp)
		atomic.Cas(&t.status, timerRunning, timerNoStatus)
	}

	unlock(&pp.timersLock)
	f(arg, seq)
	lock(&pp.timersLock)
}

• If the period field is greater than 0
  • Modify the timer's next trigger time and update its position in the heap
  • Update the status of the timer to timerWaiting
  • Set the timer0When field of the processor
• If period is less than 0
  • Delete timer
  • Update timer status to timerNoStatus

4, Trigger timer

The timer of Go will be triggered in the following two modules

• When the scheduler schedules, it checks whether the timer in the processor is ready
• System monitoring checks for unexecuted expiration counters

1. Scheduler

The scheduler uses runtime Checktimers to run the timer of the processor

func checkTimers(pp *p, now int64) (rnow, pollUntil int64, ran bool) {
	// Adjust timer
  if atomic.Load(&pp.adjustTimers) == 0 {
		next := int64(atomic.Load64(&pp.timer0When))
		if next == 0 {
			return now, 0, false
		}
		if now == 0 {
			now = nanotime()
		}
		if now < next {
			if pp != getg().m.p.ptr() || int(atomic.Load(&pp.deletedTimers)) <= int(atomic.Load(&pp.numTimers)/4) {
				return now, next, false
			}
		}
	}

	lock(&pp.timersLock)
	adjusttimers(pp)
  
  // Run timer
  rnow = now
	if len(pp.timers) > 0 {
		if rnow == 0 {
			rnow = nanotime()
		}
		for len(pp.timers) > 0 {
			if tw := runtimer(pp, rnow); tw != 0 {
				if tw > 0 {
					pollUntil = tw
				}
				break
			}
			ran = true
		}
	}
  
  // Delete timer
  if pp == getg().m.p.ptr() && int(atomic.Load(&pp.deletedTimers)) > len(pp.timers)/4 {
		clearDeletedTimers(pp)
	}

	unlock(&pp.timersLock)
	return rnow, pollUntil, ran

• Adjust timer
  • If there is no timer to adjust in the processor
    • Returns directly when there is no timer to execute
    • Return directly when the next timer has not expired and there are few timers to delete (less than a quarter of the total)
  • If there is a timer in the processor that needs to be adjusted, run time is called adjusttimers
• Run timer
  • If there is a timer to be executed, run it directly
  • If it does not exist, get the trigger event of the latest timer
• Delete timer
  • If the processor P of the current goroutine is the same as the incoming processor, and the timer to be deleted in the processor is more than 1 / 4 of the timer in the heap, the timer to be deleted will be deleted

2. System monitoring

The system monitoring function may also trigger the timer of the function

func sysmon() {
	...
	for {
		...
		now := nanotime()
		next, _ := timeSleepUntil()
		...
		lastpoll := int64(atomic.Load64(&sched.lastpoll))
		if netpollinited() && lastpoll != 0 && lastpoll+10*1000*1000 < now {
			atomic.Cas64(&sched.lastpoll, uint64(lastpoll), uint64(now))
			list := netpoll(0)
			if !list.empty() {
				incidlelocked(-1)
				injectglist(&list)
				incidlelocked(1)
			}
		}
		if next < now {
			startm(nil, false)
		}
		...
}

• Gets the expiration time of the timer and the heap that holds the timer
  • This will traverse all the timers and find the next timer to execute
• If there is no polling for more than 10ms, start network polling
• If a timer is not executed and the processor cannot be preempted, a new thread processing timer should be started at this time