With the development of Javascript language, ES6 specification has brought us many new contents, among which Generators is an important feature. Using this feature, we can simplify the creation of iterators. What's more exciting is that Generators allow us to pause during function execution and resume execution at some point in the future. This feature changes the previous feature that functions must be executed before they return. Applying this feature to asynchronous coding can effectively simplify the writing of asynchronous methods and avoid falling into callback hell.
This paper will briefly introduce Generators, and then focus on the operating mechanism of Generators and its implementation principle in ES5 combined with the author's experience in C#.

1. Brief introduction to Generators

A simple Generator function example

function* example() {
  yield 1;
  yield 2;
  yield 3;
}
var iter=example();
iter.next();//{value:1，done:false}
iter.next();//{value:2，done:false}
iter.next();//{value:3，done:false}
iter.next();//{value:undefined，done:true}

The above code defines a generator function. When the generator function example() is called, it does not execute the function immediately, but returns a generator object. Whenever the. next() method of the generator object is called, the function runs to the next yield expression, returns the expression result and pauses itself. When the end of the generator function is reached, the value of done in the returned result is true and the value is undefined. We call the above example() function as a generator function. Compared with ordinary functions, the two have the following differences

• Ordinary functions are declared with function, and generator functions are declared with function *
• Normal functions use the return value, and generator functions use the yield return value
• The normal function is in the run to completion mode, that is, after the normal function starts execution, it will be executed until all statements of the function are completed. During this period, other code statements will not be executed; The generator function is in the run pause run mode, that is, the generator function can be suspended one or more times during the operation of the function, and the execution can be resumed later. During the suspension, other code statements are allowed to be executed

This article will not introduce the use of Generators. For more information, it is recommended to read the following series of articles, <ES6 Generators: Complete Series> perhaps Deep understanding of ECMAScript 6 asynchronous programming Series articles
2.Generators in C#
Generator is not a new concept. I first came into contact with this concept when learning to use c#. C # introduced the yield keyword from version 2.0, which makes it easier to create enumerators and enumerable types. The difference is that it is not named Generators in c# but iterators.
This article will not introduce the enumerable class IEnumerable and enumerator IEnumerator in C#. For information, it is recommended to read the relevant chapters of C#4.0 graphical tutorial.
2.1 introduction to c# iterators
Let's take a look at an example. The following method declaration implements an iterator that generates and returns enumerators

public IEnumerable <int> Example()
{
		yield return 1;
		yield return 2;
		yield return 3;
}

The method definition is very close to the ES6 Generators definition. The definition declares that a generic enumerable type of int type is returned. The method body returns the value through the yield return statement and suspends its execution.
Use iterators to create classes of enumerable types

class YieldClass
{
    public IEnumerable<int> Example()//iterator 
    {
	yield return 1;
	yield return 2;
	yield return 3;
    }
}
class Program
{
    static void Main()
    {
	YieldClass yc=new YieldClass ();
	foreach(var a in yc.Example())
		Console.WriteLine(a);
    }
}

The above code will produce the following input

1
2
3

2. Principle of C# iterator

In. Net, yield is not a feature of. Net runtime, but a syntax sugar. When the code is compiled, this syntax sugar will be compiled into simple IL code by C# compiler.
Continuing to study the above example, we can see from the Reflector decompiler that the compiler generates an internal class for us with the following declaration

[CompilerGenerated]
private sealed class YieldEnumerator : 
   IEnumerable<object>, IEnumerator<object>
{
    // Fields field
    private int state;
    private int current;
    public YieldClass owner;
    private int initialThreadId;
 
    // Methods method
    [DebuggerHidden]
    public YieldEnumerator(int state);
    private bool MoveNext();
    [DebuggerHidden]
    IEnumerator<int> IEnumerable<int>.GetEnumerator();
    [DebuggerHidden]
    IEnumerator IEnumerable.GetEnumerator();
    [DebuggerHidden]
    void IEnumerator.Reset();
    void IDisposable.Dispose();
 
    // Properties property
    object IEnumerator<object>.Current 
    { [DebuggerHidden] get; }
 
    object IEnumerator.Current 
    { [DebuggerHidden] get; }
}

The original Example() method returns only one instance of YieldEnumerator and passes the initial state - 2 to itself and its references. Each iterator holds a state indication

• -2: Initialize to iteratable class Enumerable
• -1: End of iteration
• 0: initialize as iterator Enumerator
• 1-n: yield return index value in the original Example() method

The code in the Example() method is converted to YieldingEnumerator.MoveNext(). In our example, the converted code is as follows

bool MoveNext()
{
    switch (state)
    {
        case 0:
            state = -1;
            current = 1;
            state = 1;
            return true;
        case 1:
            state = -1;
            current = 2;
            state = 2;
            return true;
        case 2:
            state = -1;
            current = 3;
            state = 3;
            return true;
        case 3:
            state = -1;
            break;
    }
    return false;
}

Using the above code conversion, the compiler generates a state machine for us. It is based on this state machine model that realizes the characteristics of yield keyword.
The iterator state machine model can be shown in the following figure

• Before is the initial state of the iterator
• Running is to call MoveNext and enter this state. In this state, the enumerator detects and sets the location of the next item. Exit this status when you encounter yield return, yield break or the end of the iteration
• Suspended is the state in which the state machine waits for the next call to MoveNext
• After is the status of the end of the iteration

3,Generators in Javascript

By reading the above, we learned about the use of Generator in C# and by looking at the IL code generated by the compiler, we learned that the compiler will generate an internal class to save the context information, then convert the yield return expression into switch case, and realize the characteristics of yield keyword through state machine mode.
3.1 principle analysis of JavaScript generators
How to implement the yield keyword in Javascript?
First, the generator is not a thread. In languages that support threads, multiple pieces of different code can run at the same time, which often leads to resource competition. If used properly, there will be a good performance improvement. The generator is completely different. The Javascript execution engine is still a single threaded environment based on event loop. When the generator runs, it will run in the same thread called caller. The order of execution is orderly and definite, and there will never be concurrency. Unlike threads in the system, generators are suspended only when they use yield internally.
Since the generator is not supported by the engine from the bottom, we can follow the experience of exploring the principle of yield feature in C# above, regard the generator as a syntax sugar, and use an auxiliary tool to convert the generator function into ordinary Javascript code. In the converted code, there are two key points: one is to save the context information of the function, The second is to implement a perfect iterative method to make multiple yield expressions execute in order, so as to realize the characteristics of the generator.

4, How Generators work in ES5

Regenerator The tool has realized the above ideas. With the help of the Regenerator tool, we can use the generator function in the native ES5. In this section, we will analyze the implementation of the Regenerator to deeply understand the operation principle of Generators.
adopt This online address You can easily view the converted code, still taking the initial article as an example

function* example() {
  yield 1;
  yield 2;
  yield 3;
}
var iter=example();
iter.next();

Converted to

var marked0$0 = [example].map(regeneratorRuntime.mark);
function example() {
  return regeneratorRuntime.wrap(function example$(context$1$0) {
    while (1) switch (context$1$0.prev = context$1$0.next) {
      case 0:
        context$1$0.next = 2;
        return 1;
 
      case 2:
        context$1$0.next = 4;
        return 2;
 
      case 4:
        context$1$0.next = 6;
        return 3;
 
      case 6:
      case "end":
        return context$1$0.stop();
    }
  }, marked0$0[0], this);
}
var iter = example();
iter.next();

As can be seen from the converted code, similar to the conversion of yield return expression by C# compiler, the Regenerator rewrites the yield expression in the generator function as switch case, and uses context context $1 $0 in each case to save the current context state of the function.
In addition to switch case, the iterator function example is wrapped by regeneratoruntime.mark and returns an iterator object wrapped by regeneratoruntime.wrap.

runtime.mark = function(genFun) {
  if (Object.setPrototypeOf) {
    Object.setPrototypeOf(genFun, GeneratorFunctionPrototype);
  } else {
    genFun.__proto__ = GeneratorFunctionPrototype;
  }
  genFun.prototype = Object.create(Gp);
  return genFun;
};

Wrap example into the following objects through mark wrapping

When the generator function example() is called, an iterator object wrapped by the wrap function is returned

runtime.wrap=function (innerFn, outerFn, self, tryLocsList) {
  // If outerFn provided, then outerFn.prototype instanceof Generator.
  var generator = Object.create((outerFn || Generator).prototype);
  var context = new Context(tryLocsList || []);
 
  // The ._invoke method unifies the implementations of the .next,
  // .throw, and .return methods.
  generator._invoke = makeInvokeMethod(innerFn, self, context);
 
  return generator;
}

The iterator object returned is as follows

When the iterator object iter.next() method is called, the following code will be executed_ invoke method. According to the wrap method code above, the makeInvokeMethod (innerFn, self, context) of the iterator object is finally called; method

// Helper for defining the .next, .throw, and .return methods of the
// Iterator interface in terms of a single ._invoke method.
function defineIteratorMethods(prototype) {
  ["next", "throw", "return"].forEach(function(method) {
    prototype[method] = function(arg) {
      return this._invoke(method, arg);
    };
  });
}

The makeInvokeMethod method contains a lot of content, so some analysis is selected here. First, we found that the generator initializes its state to "Suspended Start"

function makeInvokeMethod(innerFn, self, context) {
  var state = GenStateSuspendedStart;
 
  return function invoke(method, arg) {

makeInvokeMethod returns the invoke function. When we execute the. next method, we actually call the following statements in the invoke method

var record = tryCatch(innerFn, self, context);

Here, fn in the tryCatch method is the converted example $method and arg is the context object context. Because the reference to the context inside the invoke function forms a closure reference, the context context can be maintained during the iteration.

function tryCatch(fn, obj, arg) {
  try {
    return { type: "normal", arg: fn.call(obj, arg) };
  } catch (err) {
    return { type: "throw", arg: err };
  }
}

The tryCatch method will actually call the example $method, enter the converted switch case, and execute the code logic. If the result is a normal type value, we wrap it in an iteratable object format and update the generator state to GenStateCompleted or genstatesuspended year

var record = tryCatch(innerFn, self, context);
        if (record.type === "normal") {
          // If an exception is thrown from innerFn, we leave state ===
          // GenStateExecuting and loop back for another invocation.
          state = context.done
            ? GenStateCompleted
            : GenStateSuspendedYield;
 
          var info = {
            value: record.arg,
            done: context.done
          };

5. Summary

By analyzing the generator code and tool source code converted by Regenerator, we explore the operation principle of the generator. The Regenerator wraps the generator function through the tool function and adds methods such as next/return to it. At the same time, the returned generator object is wrapped, so that the calls to methods such as next finally enter the state machine model composed of switch case. In addition, use the closure technique to save the context information of the generator function.
The above process is basically consistent with the implementation principle of yield keyword in C# and adopts the idea of compilation and transformation, uses the state machine model, and saves the function context information at the same time. Finally, the new language features brought by the new yield keyword are realized.
​

6. Reference articles

​

1.ES6 Generators:Complete Series articles
2. ES6 Generators
3.In depth understanding of ECMAScript 6 asynchronous programming series articles
4.ES6 Generators:How do they work?
5.Behind the scenes of the C# yield keyword