First experience of front-end recording and playback

Posted by simongcc on Sat, 19 Feb 2022 03:34:45 +0100

1. Problem background

1.1 what is front-end recording and playback?

As the name suggests, it is to record various operations of users in the web page, and support playback operations at any time.

1.2 why?

When it comes to needs, we have to say a classic scenario. Generally, the front end does exception monitoring and error reporting. It will collect and report JavaScript error reporting information and other relevant data in the process of website interaction in the form of self-development or access to a third-party SDK, that is, embedding points.

In the traditional buried point scheme, the specific error reporting code file and row and column information can be located according to SourceMap. It can basically locate most of the scene problems, but in some cases it is difficult to reproduce the errors. Most of them are one of the programmer's mantras during the test wrangling (I didn't report an error here, is there a problem with your computer).

If only we could record the wrong operation process, it would be convenient for us to reproduce the scene and retain the evidence, as if we had dug a hole for ourselves.

Add: I am a reprint. Our demand is to replay the whole insurance process of users (our company is an insurance company)

1.3 how to achieve?

Can the front end record video? My first reaction was to question, and then I was a wave of Google and found that there was a feasible solution.

Before Google, I thought of setting a timer to take a screenshot of the view window. The screenshot can be realized by canvas 2html, but this method will undoubtedly cause performance problems and will be rejected immediately.

Here's what I "know" about Google's solution. If you have any questions, please correct them.

2. Initial ideas

Web page is essentially a DOM node, which is rendered by browser. Whether we can save the DOM in some way and continuously record the DOM data state at different time nodes. Then restore the data to DOM node and render it to complete playback?

2.1 operation record

Through document documentElement. CloneNode () is a data object cloned into DOM. At this time, this data cannot be directly transmitted to the back end through the interface. Some formatting preprocessing needs to be carried out to process it into a data format convenient for transmission and storage. The simplest way is to serialize, that is, convert to JSON data format.

let docJSON = {
  "type": "Document",
  "childNodes": [
      "type": "Element",
      "tagName": "html",
      "attributes": {},
      "childNodes": [
          "type": "Element",
          "tagName": "head",
          "attributes": {},
          "childNodes": []

After having complete DOM data, you also need to listen when the DOM changes and record the DOM node information of each change. To monitor data, you can use MutationObserver, which is an API that can monitor DOM changes.

const observer = new MutationObserver(mutationsList => {
    console.log(mutationsList); // Changed data
// Start observing the target node with the above configuration
observer.observe(document, {});

In addition to monitoring DOM changes, there is also event monitoring. Users interact with web pages mostly through input devices such as mouse and keyboard. Behind these interactions is JavaScript event monitoring. Event monitoring can be completed by binding system events, which also need to be recorded. Take mouse movement as an example:

// Mouse movement
document.addEventListener('mousemove', e => {
  // The pseudo code obtains the information of mouse movement and records it
    x: clientX,
    y: clientY,
    timeOffset: - timeBaseline,

Note: addEventListener can bind multiple same events without affecting the developer's event binding

2.2 playback operation

The data already exists, and then there is playback. Playback essentially restores JSON data into DOM nodes and renders them. It's not so easy to restore data!

2.3 rendering environment

First of all, in order to ensure code isolation during playback, a sandbox environment is required, which can be achieved by iframe tag, and iframe provides a sandbox attribute to configure sandbox. The function of sandbox environment is to ensure that the code is safe and undisturbed.

<iframe sandbox srcdoc></iframe>

Sandbox attribute can be used as a sandbox. Click to view the document

srcdoc can be set directly as a piece of html code

2.4 data restore

Snapshot reorganization is mainly the reorganization of DOM nodes, which is a bit like the process of transforming virtual DOM into real document nodes, but event type snapshots do not need reorganization.

2.5 timer

With data and environment, timers are also needed. Rendering DOM continuously through the timer is essentially an effect of playing video, and requestAnimationFrame is the most appropriate.

requestAnimationFrame will be executed in each frame to avoid congestion, which is a more appropriate choice than setTimeout

So far, I have a general idea that it is still a distance from landing. Thanks to open source, we can go to Github to see if there is a suitable wheel to copy (learn from), and there is just a ready-made framework rrweb might as well have a look together.

3. rrweb framework

rrweb is a front-end recording and playback framework. Its full name is record and replay the web. As its name suggests, it can record and replay the operations in the web interface. Its core principle is the scheme introduced above.

3.1 composition of rrweb

rrweb consists of three parts:

  • Rrweb snapshot mainly deals with the serialization and reorganization of DOM structure;
  • The main functions of rrweb are recording and playback;
  • Rrweb player a video player UI space
3.2 rrweb usage

npm installation is common, and import/require is not a big problem

3.2.1 recording

Via rrweb Record method to record the page, and the emit callback can accept the recorded data.

// 1. Recording
let events = []; // Record snapshot

  emit(event) {
    // Store event into the events array
3.2.2 playback

Via rrweb Replay can play back the video and need to transfer the recorded data.

// 2. Playback
const replayer = new rrweb.Replayer(events);;

4. rrweb source code

According to the above ideas, I will analyze some of the key codes. Of course, it's just some analysis based on my personal understanding. In fact, rrweb source code is far more than that.

The core part consists of three blocks: record, replay and snapshot.

4.1 Record recording

After the DOM is loaded, record will do a complete DOM serialization. We call it full snapshot, which records the whole HTML data structure.

In record Find the definition init of the key entry function in TS. the entry function calls takeFullSnapshot and observe(document) functions when the document loading is completed or (interactive, complete).

if (
    document.readyState === 'interactive' ||
    document.readyState === 'complete'
) {
} else {
    on('load',() => { init(); },),
const init = () => {
    takeFullSnapshot(); // Generate full snapshot
    handlers.push(observe(document)); //monitor

document.readyState contains three states:

1. interactive;

2. loading;

3. complete

From the literal meaning, takeFullSnapshot is used to generate a "complete" snapshot, that is, it will sequence document s into a complete data, which is called full snapshot.

All serialization related operations are completed using snapshot, which accepts a dom object and a configuration object, passes document, and serializes the whole page to get the completed snapshot data.

// Generate full snapshot
takeFullSnapshot = (isCheckout = false) => {
    const [node, idNodeMap] = snapshot(document, {
        //... Some configuration items

idNodeMap is a key value key value pair object with id key and DOM object value

observe(document) is the initialization of some listeners. It also passes the whole document object to listen, and initializes some listeners by calling initObservers.

const observe = (doc: Document) => {
    return initObservers(//...)

On observer The initObservers function definition can be found in the TS file. This function initializes 11 listeners, which can be divided into three categories: DOM type / Event type / Media media:

export function initObservers(
    // dom
    const mutationObserver = initMutationObserver();
    const mousemoveHandler = initMoveObserver();
    const mouseInteractionHandler = initMouseInteractionObserver();
    const scrollHandler = initScrollObserver();
    const viewportResizeHandler = initViewportResizeObserver();
    // ...

DOM change listener mainly includes DOM change (addition, deletion and modification) and style change. The core is realized through MutationObserver

let mutationObserverCtor = window.MutationObserver;

const observer = new mutationObserverCtor(
    // Handling changing data
observer.observe(doc, {});
return observer;

Interactive monitoring - Take mouse movement initMoveObserver as an example

// Mouse movement record
function initMoveObserver() {
    const updatePosition = throttle<MouseEvent | TouchEvent>(
        (evt) => {
                x: clientX,
                y: clientY,
    const handlers = [
        on('mousemove', updatePosition, doc),
        on('touchmove', updatePosition, doc),

The media type listener includes canvas / video / audio. Taking video as an example, it essentially records the play and pause status, and mediaInteractionCb calls back the play / pause status.

function initMediaInteractionObserver(): listenerHandler {
        type: type === 'play' ? MediaInteractions.Play : MediaInteractions.Pause,
        id: mirror.getId(target as INode),

5. Snapshot

snapshot is responsible for serialization and reorganization. It mainly handles DOM serialization through serializeNodeWithId and DOM reorganization through rebuildWithSN function.

The serializeNodeWithId function is responsible for serialization and mainly does three things:

  • Call serializeNode to serialize Node;
  • Generate a unique ID through genId() and bind it to the Node;
  • The recursive implementation serializes the child nodes and finally returns an object with ID
// Serialize a DOM with ID
export function serializeNodeWithId(n) {
  // 1. serializeNode, the serialization core function
  const _serializedNode = serializeNode(n);
  // 2. Generate unique ID
  let id = genId();
  // Binding ID
  const serializedNode = Object.assign(_serializedNode, { id });
  // 3. Child node serialization recursion
  for (const childN of Array.from(n.childNodes)) {
    const serializedChildNode = serializeNodeWithId(childN, bypassOptions);
    if (serializedChildNode) {

The core of serializeNodeWithId is to serialize DOM through serializeNode, and do some special processing for different nodes.

Processing of node attributes:
for (const { name, value } of Array.from((n as HTMLElement).attributes)) {
    attributes[name] = transformAttribute(doc, tagName, name, value);

Handle the external css style, get the specific style code through getCssRulesString, and store it in attributes.

const cssText = getCssRulesString(stylesheet as CSSStyleSheet);
if (cssText) {
    attributes._cssText = absoluteToStylesheet(

To process the form, the logic is to save the selected state and do some security processing, such as replacing the content of the password box with *.

if (
    attributes.type !== 'radio' &&
    attributes.type !== 'checkbox' &&
    // ...
) {
    attributes.value = maskInputOptions[tagName] 
        ? '*'.repeat(value.length) 
        : value;
  } else if (n.checked) {
    attributes.checked = n.checked;

Save canvas status save canvas data through toDataURL

attributes.rr_dataURL = (n as HTMLCanvasElement).toDataURL();

rebuild is responsible for rebuilding the DOM:

Reorganize nodes through buildNodeWithSN function
Recursive call to reorganize child nodes

export function buildNodeWithSN(n) {
  // DOM reorganization core function buildNode
  let node = buildNode(n, { doc, hackCss });
  // Child node reconstruction and appendChild
  for (const childN of n.childNodes) {
    const childNode = buildNodeWithSN(childN);
    if (afterAppend) {

6. Replay playback

The playback part is in replay TS file, first create the sandbox environment, and then reconstruct the full snapshot of the document. Play the incremental snapshot by simulating the timer through the requestAnimationFrame.

The constructor of replay receives two parameters, snapshot data events and configuration item config

export class Replayer {
    constructor(events, config) {
        // 1. Create sandbox environment
        // 2. Timer
        const timer = new Timer();
        // 3. Playback service
        this.service = new createPlayerService(events, timer);

The core three steps in the constructor are to create a sandbox environment, a timer, and initialize the player and start it. Player creation depends on events and timer. In essence, it still uses timer to play.

6.1 sandbox environment

First, in replay This can be found in the constructor of TS The core of setupDom is to create a sandbox environment through iframe.

private setupDom() {
  // Create iframe
  this.iframe = document.createElement('iframe'); = 'none';
  this.iframe.setAttribute('sandbox', attributes.join(' '));
6.2 playback service

Also in replay In the TS constructor, the createPlayerService function is called to create the player server, which is machine. in the same directory. TS defines that the core idea is to add snapshot actions to the timer and call the timer Start() starts playback of the snapshot.

export function createPlayerService() {
    play(ctx) {
        // Get the doAction function executed by each event
        for (const event of needEvents) {
            const castFn = getCastFn(event);
                doAction: () => {
         // Add to timer queue
         // Start timer to play video

The playback service uses the third-party library @ xstate/fsm state machine to control various states (playback, pause, live broadcast)

Timer TS is also in the same level directory. The core is to realize the timer function through requestAnimationFrame, play back the snapshot, store the snapshot actions to be played in the form of queue, and then call action recursively in start DoAction to realize the snapshot restore of the corresponding time node.

export class Timer {
    // Add queue
    public addActions(actions: actionWithDelay[]) {
        this.actions = this.actions.concat(actions);
    // Play queue
    public start() {
        function check() {
            // ...
            // Loop call doAction in actions, that is, the castFn function
            while (actions.length) {
                const action = actions[0];
                // doAction will play back the snapshot and perform different actions for different snapshots
            if (actions.length > 0 || self.liveMode) {
                self.raf = requestAnimationFrame(check);
        this.raf = requestAnimationFrame(check);

doAction will perform different actions in different types of snapshots. In the playback service, doAction will eventually call the getCastFn function to do some case s:

private getCastFn(event: eventWithTime, isSync = false) {
    switch (event.type) {
        case EventType.DomContentLoaded: //dom load parsing completed
        case EventType.FullSnapshot: // Full snapshot
        case EventType.IncrementalSnapshot: //increment
            castFn = () => {
                this.applyIncremental(event, isSync);

The applyIncremental function performs different processing for different incremental snapshots, including DOM increment, mouse interaction, page scrolling, etc. taking the case of DOM incremental snapshot as an example, it will eventually go to applyMutation:

private applyIncremental(){
  switch (d.source) {
      case IncrementalSource.Mutation: {
        this.applyMutation(d, isSync); // DOM change
      case IncrementalSource.MouseMove: //Mouse movement
      case IncrementalSource.MouseInteraction: //Mouse click event

applyMutation is the place where the DOM restore operation is finally performed, including the addition, deletion and modification steps of DOM:

private applyMutation(d: mutationData, useVirtualParent: boolean) {
    d.removes.forEach((mutation) => {
        //.. Remove dom
    const appendNode = (mutation: addedNodeMutation) => {
        // Add dom to specific node
    d.adds.forEach((mutation) => {
        // add to
    d.texts.forEach((mutation) => {
        //... text processing 
    d.attributes.forEach((mutation) => {
        //... Attribute processing

The above is the key process implementation code of playback. rrweb not only does this, but also includes data compression, mobile terminal processing, privacy issues and other details. If you are interested, you can check the source code yourself.


This idea of recording and playback is really worth learning. The process of reading rrweb source code also benefits a lot. Some uses of data structures in the source code, such as double linked list, queue, tree, etc., are also worth a list.

Reference articles
  • rrweb-io/rrweb
  • rrweb: open the black box of web page recording and playback