Use of writable streams

The highWaterMark of the writable stream indicates how many values the file is expected to accept.



end not only writes, but also triggers the close event.



A true and a false is because our highWaterMark is set to 3. We want to use only 3 memory to write, but the returned value has nothing to do with whether we write or not. If false is returned, it will also be written.

• But there is a problem. When we write multiple wirte, it is a concurrent asynchronous operation, so we can't determine which is fast and which is slow.

• You can turn concurrent asynchronous operations into serial asynchronous operations.

• In addition to the first write, the next write is queued. After the first write is completed, each write in the queue is taken out for execution. It's a bit like EventLoop. Until the queue is emptied, but the queue cache may be too large, so you need an expectation, that is, highWaterMark to control it. After reaching the expectation, don't call the write method. Although it will be written in again.

• Combine fs.createStream.
  
  When we write for the first time, when the file can't eat, that is, when the value of highWaterMark is reached, we should stop writing. When the file is finished, the drain method is triggered, and then restore rs.resume(). At this time, the subsequent writes are written to the real file, rather than waiting in line all the time. The default highWaterMark for reading is 64k, while the default highWaterMark for writing is 16k.

• This is very different from the first type of concurrent asynchronous write.

Implementation of writable stream

Writable streams are implemented based on linked lists. Because writable streams involve queue sorting, they often use the addition and deletion of headers. The efficiency of adding and deleting the head and tail of the linked list is relatively high.
The implementation of linked list can be viewed as follows: Single linked list,Double linked list

• The Readable stream is internally implemented based on steam and stream's Readable (similar to our myreadsteam class, which calls fs.read) and events. fs implements ReadStream, inherits the Readable of stream, and implements its own_ read method.
• The internal of Writable stream is implemented based on stream, Writable and events. The writiesstream implemented by fs inherits the Writable of stream. Self realized_ The wirte method is called by the wirte method of the parent class.

To achieve ten numbers, I want to use three memory to process

Native.

Implement your own WriteStream

Idea: like the readable stream, the open and wirte operations are separated, and the event publishing mode is adopted. Then judge whether the current write is the first call through the variable to ensure that only one write is executed at a time, and all the others are thrown into the cache. When the write is completed, take it out of the cache one by one to execute the write. Until the cache is emptied, the drain event is triggered to notify the user that the cache is cleared and the variable is set to initialization.
This enables concurrent asynchronous operations to become serial asynchronous operations.
Initialization variables are somewhat different from ReadStream. For example, there is no end, encoding is written to utf8 by default, and so on.

len is used to judge the length of the current cached value, and needDrain is used to judge whether there are too many caches and whether they meet the expected value. cahce is a cache queue. writing is used to identify whether to write for the first time.

• open method
  
• Execute the write method
  
  Because the data may be in Chinese and English, the first step is to change to buffer by default. Then judge whether the length of the buffer written by the current write + this.len (the length of the cache queue) exceeds the expected value highWaterMark.
  Rewrite the cb function so that each time cb is successfully called, the clearBuffer method will be executed.
  Then judge whether to write for the first time through writing, and call it for the first time_ Use the write method to execute fs.write, otherwise it will be put into the cache.
• _ write method
  
  The write method may not be open when the user executes it. Because it is asynchronous, it needs to be processed, and then execute the write method to write the content. After each write, maintain the offset and reduce len. Execute cb(). cb is rewritten and will call the clearBuffer method.
• clearBuffer method
  
  This method mainly takes out and executes the write tasks in the cache one by one. Note that what is called here is_ The write method is to actually write to the file. After the cache is emptied, you need to start the drain event to tell the user that it has been emptied and that you can continue to write. And reset some variables.
  
  
  Four drain events are triggered and four are successfully written.

Summary:

Through the cache, global variables, and event system, only the first write will actually execute fs.write, and other write methods will be put into the cache. Only after fs.write is executed, will it continue to fetch from the cache for fs.write. Until the cache is empty, the drain event is triggered to reset the variable so that the user can continue to call the write method to write. Just like when a person eats a meal and feeds a large one, he always eats a small one first, chews the rest in his mouth, and returns false to tell you, don't feed it first, wait until all is finished. When it slowly chews and swallows the current oral food, it will tell you that it can continue to feed, and so on until the bowl of rice is eaten (all written)

Optimize using linked list instead of array cache

Implementation of linked list: in Single linked list here.
experiment:


The first 3 is deleted successfully, and the linked list is 6 = > 10.

The transformation is completed,
It is expected to be 3, so write will be called three times for the first time, and two writes will be stored in the queue, as shown in

First, there are two in the queue, and then after 49 is taken out, there is a 50 left, and then it is empty. Done.
All codes:

// WriteStream
const LinkedList = require("./linklist");

//Implementation of queue based on linked list
class Queue {
  constructor() {
    this.link = new LinkedList();
  }
  //Add in the last one
  offer(element) {
    this.link.append(element);
  }
  //Remove the first bit of the linked list and return
  shift() {
    return this.link.removeAt(0);
  }
}

class WriteStream extends EventMitter {
  constructor(path, options) {
    super();
    this.path = path;
    this.flags = options.flags || "w";
    this.encoding = options.encoding || "utf8";
    this.autoClose = options.autoClose || true;
    this.mode = options.mode || 0o666;
    this.start = options.start || 0;
    this.highWaterMark = options.highWaterMark || 16 * 1024;
    this.emitClose = options.emitClose || true;
    this.offset = this.start; // Displacement per write to file

    this.fd = undefined;

    this.len = 0; //Judgment cache
    this.needDrain = false; //Need to trigger drain
    this.cache = []; //Second start write cache
    this.writing = false; //Identifies whether writing is in progress
    this.queue = new Queue();
    this.open();
  }

  open() {
    fs.open(this.path, this.flags, this.mode, (err, fd) => {
      this.fd = fd;
      this.emit("open", fd);
    });
  }

  _write(chunk, encoding, cb) {
    if (typeof this.fd !== "number") {
      this.once("open", () => {
        this._write(chunk, encoding, cb);
      });
      return;
    }
    fs.write(this.fd, chunk, 0, chunk.length, this.offset, (err, written) => {
      this.offset += written; //Maintain offset
      this.len -= written; //Reduce the number of caches
      cb(); //Callback
    });
  }

  //Take out the cache queue waiting tasks in turn and write them one by one.
  clearBuffer() {
    const a = JSON.stringify(this.queue);
    console.log(a);
    const data = this.queue.shift();
    if (data) {
      this._write(data.chunk, data.encoding, data.cb);
    } else {
      //After reading the cache, the drain event needs to be triggered
      this.writing = false; //The current write has completed.
      if (this.needDrain) {
        //If too much is deposited, the expected value is too large
        this.needDrain = false;
        this.emit("drain");
      }
    }
  }
}

//Simulate the write method on the Writable instance of the Stream
EventMitter.prototype.write = function (
  chunk,
  encoding = this.encoding,
  cb = () => {}
) {
  // Asynchronous method
  // Pay attention to the comparison between Chinese and English. Convert all data to Buffer
  chunk = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
  this.len += chunk.length;

  //Did the judgment meet expectations
  const returnValue = this.len < this.highWaterMark ? true : false;

  //After data is written, drain needs to be triggered and Len needs to be subtracted
  this.needDrain = !returnValue;

  //  Clear the logic of the cache queue
  const userCb = cb;
  cb = () => {
    userCb();
    this.clearBuffer();
  };

  //Judge whether to input the data for the first time, because subsequent write s are directly put into the cache
  if (!this.writing) {
    //For the first write, the write operation is actually performed
    this._write(chunk, encoding, cb);
    this.writing = true;
  } else {
    //The second write is saved in the cache
    this.queue.offer({
      chunk,
      encoding,
      cb,
    });
  }
  return returnValue;
};

const ws = new WriteStream("b.txt", {
  flags: "w",
  encoding: null,
  autoClose: true,
  start: 0, //There is no end attribute, only start
  highWaterMark: 3, //Unlike a readable stream, the highWaterMark of a writable stream indicates that the file is expected to accept only three memories
});

ws.on("open", () => {
  console.log("file open");
});

let i = 0;
function write() {
  let flag = true;
  while (i < 4 && flag) {
    flag = ws.write(i++ + "");
    console.log('flag', flag);
  }
}

ws.on("drain", () => {
  //The drain event is triggered only after the inhaled data reaches the expected value and the data has been written to the file.
  console.log("Finished");
  write();
});

write();

// Linked list
class Node {
  constructor(data) {
    this.data = data;
    this.next = null;
  }
}
module.exports = class LinkedList {
  constructor() {
    //Head pointer
    this.head = null;
    //Length of linked list
    this.length = 0;
  }

  append(data) {
    const element = new Node(data);
    if (!this.head) {
      this.head = element;
    } else {
      let current = this.head;

      //Traverse to find the last node
      while (current.next) {
        current = current.next;
      }
      //insert
      current.next = element;
    }
    this.length++;
  }

  // //Insert at a specific location
  insert(position, data) {
    if (
      typeof position !== "number" ||
      position < 0 ||
      position > this.length
    ) {
      return false;
    }
    const element = new Node(data);
    //Insert first
    if (position === 0) {
      element.next = this.head;
      this.head = element;
    } else {
      //Insert the middle bit
      let current = this.head;
      let index = 1;
      //Use the previous bit of the insertion position for operation. For example, insert into the fourth, point element.next to the next of the third, and then re point the next of the third to element
      while (index++ < position) {
        // For example, when position = 4 and index = 4, current points to the third because it is executed twice
        current = current.next;
      }
      // Make element the fourth
      element.next = current.next;
      current.next = element;

      // for (let i = 1; i < position; i++) {
      //     if (i === position - 1) {
      //         element.next = current.next
      //         current.next = element
      //         break;
      //     }
      //     current = current.next
      // }
    }
    this.length++;
  }

  // //Get the element of the corresponding position
  get(position) {
    if (
      typeof position !== "number" ||
      position < 0 ||
      position >= this.length
    ) {
      return undefined;
    }
    let current = this.head;
    let index = 0;
    while (index++ < position) {
      current = current.next;
    }
    return current.data;
  }

  // //Returns the index of the element in the list
  indexOf(data) {
    let current = this.head;
    let index = 0;
    while (current) {
      if (current.data === data) {
        return index;
      }
      current = current.next;
      index++;
    }
    return -1;
  }

  // //Modify an element at a location
  update(position, data) {
    if (
      typeof position !== "number" ||
      position < 0 ||
      position >= this.length
    ) {
      return false;
    }
    let current = this.head;
    let index = 0;
    while (index++ < position) {
      current = current.next;
    }
    current.data = data;
  }

  // //Removes an item from a specific location in the list
  removeAt(position) {
    if (
      typeof position !== "number" ||
      position < 0 ||
      position >= this.length
    ) {
      return undefined;
    }
    let current = this.head;
    if (position === 0) {
      const headElData = this.head.data;
      this.head = this.head.next;
      this.length--;
      return headElData;
    }
    let index = 0;
    // Find the previous node that should be deleted, such as deleting 2. At this time, the current is the node pointed to by 1.
    while (index++ < position - 1) {
      current = current.next;
    }
    const currentElement = current.next;
    let element = current.next.next; //Save the third node
    current.next.next = null; //Break the relationship between the second node and the third node
    current.next = element; // The first node points to the third node
    this.length--;
    return currentElement.data;
  }

  // Remove an item from the list
  remove(element) {
    let current = this.head;
    let nextElement;
    let isSuccess;

    //If the first one is
    if (current.data === element) {
      nextElement = this.head.next;
      this.head = element;
      element.next = nextElement;
      return true;
    }

    // Find the previous node that should be deleted, such as deleting 2. At this time, the current is the node pointed to by 1.
    while (current) {
      if (current.next.data === element) {
        nextElement = current.next.next; //Save next node
        current.next.next = null; //Disconnect the next node from the next node
        current.next = nextElement; //Connect next node
        isSuccess = true;
        break;
      } else {
        current = current.next;
      }
    }

    //Delete successfully
    if (isSuccess) {
      this.length--;
      return true;
    }

    return false;
  }

  isEmpty() {
    return !!this.length;
  }

  size() {
    return this.length;
  }

  toString() {
    let current = this.head;
    let str = "";
    while (current) {
      str += `,${current.data.toString()}`;
      current = current.next;
    }
    return str.slice(1);
  }
};