Chapter VII network programming

Node provides four modules: net, dgram, HTTP and HTTPS, which are used to process TCP, UDP, HTTP and HTTPS respectively. They are suitable for clients and servers.

TCP

TCP transmission control protocol belongs to the transport layer in the OSI model. Many application layer protocols are built based on TCP, such as HTTP, SMTP, IMAP, etc. Review the OSI model.

• Layer 7: the application layer provides an interface for the operating system or network application to access network services. Representatives of application layer protocols include HTTP, HTTPS, FTP, TELNET, SSH, SMTP, POP3, etc.
• Layer 6: the presentation layer converts data into content compatible with the receiver and suitable for transmission, such as data encryption, compression, format conversion, etc.
• Layer 5: the session layer is responsible for setting and maintaining the communication connection between network devices in data transmission. Manage the session process between hosts, and insert check points into the data to realize data synchronization.
• Layer 4: the transmission layer adds the transmission header to the data to form a data packet to complete end-to-end data transmission. The transmission header contains protocol and other information, such as TCP, UDP, etc.
• Layer 3: the network layer is responsible for addressing and routing packets between subnets, and can also realize congestion control, Internet Interconnection and other functions. Network layer protocols include: IP, IPX, etc.
• Layer 2: the data link layer provides reliable transmission on unreliable physical media, mainly including physical address addressing, data encapsulation into frames, flow control, data verification, retransmission, etc.
• Layer 1: the physical layer transmits data frames on the LAN and is responsible for the interworking between computer communication equipment and network media, including pin, voltage, cable specification, hub, network card, host adaptation, etc.

TCP is a connection oriented protocol, which is characterized by three handshakes before transmission. Only when a session is established can the server and the client send data to each other. In the process of establishing a session, the server and the client provide a socket respectively, and the two sockets form a connection together. The connection between the server and the client is realized through socket.

Create TCP server

var net = require('net')
var server = net.createServer(function (socket) {
  // New connection
  socket.on('data', function () {
    socket.write('Hello')
  })
  // Disconnect
  socket.on('end', function () {
    console.log('Socket end')
  })
  socket.write('Welcome')
})
server.listen(8124, function () {
  console.log('server bound')
})

Use the telnet tool as the client to connect to the server just created.

$ telnet 127.0.0.1 8124
// Enter any character at will
$ Hello

Similarly, we can also listen to the Domain Socket

server.listen('/tmp/echo.sock')

The client is built by the net module for conversation. client.js:

var net = require('net')
var client = net.connect({ port: 8124 }, function() {
  //'connect' listener
  console.log('client connected')
  client.write('world!\r\n')
})
client.on('data', function(data) {
  console.log(data.toString())
  client.end()
})
client.on('end', function() {
  console.log('client disconnected')
})

Note that if it is a Domain Socket, you can fill in path when filling in options.

var client = net.connect({path: '/tmp/echo.sock'})

Events for TCP services

The above codes are divided into server events and connection events.

(1) Server events

For the server created by net.createServer(), it is an EventEmitter instance. Its custom events are as follows.

• listening: triggered after calling server.listen() to bind a port or Domain Socket. It can be written as server.listen(port, listeningListener).
• connection: triggered when each client socket connects to the server, which can be written as net.createServer().
• close: triggered when the server shuts down. server.close() will stop accepting new socket s, but save existing connections and trigger after all connections are disconnected.
• error: triggered when an exception occurs in the server.

(2) Connection event

The server can save connections with multiple clients, and each connection is a typical readable and writable Stream object. Its custom events are as follows.

• Data: when one end calls write() to send data, the other end triggers the data event.
• End: this event is triggered when either end of the connection sends FIN data.
• connect: triggered when the client socket is successfully connected to the server.
• drain: rain is strongly associated with the return value of socket.write(). When write() is called at either end, this event will be triggered at the current end.
• error: triggered when an exception occurs.
• close: triggered when the socket is closed.
• timeout: this event is triggered when the connection is no longer active for a certain time to notify the user that the current connection is idle.

TCP socket is a readable and writable Stream object, and pipe() can be used to realize pipeline operation. The following code implements the echo server.

var net = require('net')
var server = net.createServer(function(socket) {
  socket.write('Echo server\r\n')
  socket.pipe(socket)
})
server.listen(1337, '127.0.0.1')

TCP has a certain optimization strategy for small packets in the network: Nagle algorithm, which is used to reduce small packets in the network. For this situation, Nagle algorithm requires a certain amount of buffer data or a certain time before it is sent out, and Nagle algorithm combines small data packets to optimize the network at one time. However, data transmission may be delayed.

The Nagle algorithm is enabled by default in Node. You can call socket.setNoDelay(true) to close the Nagle algorithm, so that write() can immediately send data to the network.

Building UDP services

UDP, also known as user packet service, belongs to the network transport layer like TCP. UDP is not connection oriented. Once a connection is established in TCP, all sessions are completed based on the connection. If the client wants to communicate with another TCP service, it needs to create another socket for processing. In UDP, a socket can communicate with multiple UDP services.

UDP provides unreliable transmission service oriented to things. In case of poor network, there is a problem of packet loss, but it does not need connection, low resource consumption, fast and flexible processing. fico is suitable for scenes where occasionally losing one or two packets will not cause problems, such as audio, video, etc. DNS service is implemented based on UDP.

Create UDP socket

UDP socket can be used as both server and client.

var dgram = require('dgram')
var socket = dgram.createSocket('upd4')

(1) Create UDP server

Create a UDP server by calling the dgram.bind(port, [address]) method to receive network messages.

var dgram = require('dgram')
var server = dgram.createSocket('udp4')
server.on('message', function(msg, rinfo) {
  console.log('server got: ' + msg + ' from ' +
    rinfo.address + ':' + rinfo.port)
})
server.on('listening', function() {
  var address = server.address()
  console.log('server listening ' +
    address.address + ':' + address.port)
})
server.bind(41234)

(2) Create UDP client

var dgram = require('dgram')
var message = Buffer.alloc(13, 'Hello Node.js')
var client = dgram.createSocket('udp4')
client.send(message, 0, message.length, 41234, 'localhost',
  function(err, bytes) {
    client.close()
  }
)

After the client executes, the server outputs:

$ node main.js
$ server listening 0.0.0.0:41234
$ server got: Hello Node.js from 127.0.0.1:61286

When the socket is on the client side, you can call the send() method to send a message to the network.

socket.send(buf, offset, length, port, address, [callback])

(3) UDP socket events

UDP is simpler than TCP. It is only an instance of EventEmitter, not a Stream. Its custom events are as follows:

• Message: this event is triggered when a message is received after the UDP socket listens on the network card port.
• Listening: this event is triggered when UDP starts listening.
• Close: this event is triggered when the close() method is called, and the message event is no longer triggered.
• error: this event is triggered when an exception occurs.

Building HTTP services

Both TCP and UDP belong to the network transport layer protocol. If we want to construct efficient network applications, we should start from the transport layer. However, the general use of application layer protocols can meet most of our development needs. Node provides basic HTTP and HTTPS modules for HTTP and HTTPS encapsulation.

var http = require('http')
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'})
  res.end('Hello World')
}).listen(1337, '127.0.0.1')
console.log('Server running at http://127.0.0.1:1337/')

HTTP

HTTP is built on TCP and belongs to the application layer protocol.

Use curl to view the message information of network communication.

$ curl -v http://127.0.0.1:1337
* About to connect() to 127.0.0.1 port 1337 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 1337 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:1337
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2018 15:34:30 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
Hello World
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

Message analysis:

(1) TCP triple handshake

* About to connect() to 127.0.0.1 port 1337 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 1337 (#0)

(2) The client sends a request message to the server

> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:1337
> Accept: */*
>

(3) Server response client content

< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2018 15:34:30 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
Hello World

(4) End session

* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

From the above message information, we can see the characteristics of HTTP: Based on request response, the service is implemented in a question and answer manner. Although it is based on TCP session, it has no session characteristics.

http module

The HTTP module of Node contains the encapsulation of HTTP processing. In Node, HTTP service inherits from TCP service (net module). It can maintain connections with multiple clients in the form of event driven, does not create additional threads or processes for each connection, occupies very low memory, and realizes high concurrency.

The difference between HTTP service and TCP service is that after keepalive is enabled, a TCP session can be used for multiple requests and responses. TCP services in connection and HTTP services in request. The HTTP module encapsulates the process from connection to request.

In addition, the HTTP module abstracts the reading and writing of the socket used for the connection into ServerRequest and ServerResponse objects, which correspond to request and response operations respectively. In the process of request generation, the HTTP module gets the data from the connection and calls the binary module http_ After parsing the header of the request message, the parser triggers the request event and invokes the user's business logic.

(1) HTTP request

For the read operation of TCP connection, the HTTP module encapsulates it as a ServerRequest object. Header via http_parser for parsing.

> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:1337
> Accept: */*
>

• req.method property: GET
• req.url attribute:/
• req.httpVersion attribute: 1.1 Other headers are in the regular key: Value format. After being parsed, they are placed on the req.headers attribute and passed to the business logic call.

headers: {
  'user-agent': 'curl/7.29.0',
  host: '127.0.0.7:1337',
  accept: '*/*'
}

The report style part is abstracted as a read-only stream object. If the business logic needs to read the data in the report style, the operation can only be carried out after the end of the data stream.

function (req, res) {
  var buffers = []
  req.on('data', function (trunk) {
    buffers.push(trunk)
  }).on('end', function () {
    var buffer = Buffer.concat(buffers)
    res.end('')
  })
}

(2) HTTP response

The HTTP response object encapsulates the write operation of the underlying connection and can be regarded as a writable stream object to respond to the message header information through res.setHeader() and res.writeHead().

res.writeHead(200, {'Content-Type': 'text/plain'})

The converted message is as follows:

< HTTP/1.1 200 OK
< Content-Type: text/plain

setHeader can be called many times, but the message will not be written to the connection until writeHead is called. In addition, the http module will automatically set some header information.

< Date: Mon, 04 Jun 2018 15:34:30 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<

The message style is realized by calling res.write() and res.end() methods. The difference is that res.end() will call write() to send data, and then send a signal to tell the server that the response is over.

After the response, the HTTP server may use the current connection for the next request, or close the connection. In addition, it is impossible for the server to check whether an exception occurs when processing business logic. Be sure to call res.end() to end the request at the end, otherwise the client will always be in a waiting state. Of course, the long connection between the client and the server can also be realized by delaying res.end(), but the connection must be closed at the end.

(3) Events for HTTP services

The HTTP server abstracts some events and is used by the application layer. The server is also an EventEmitter instance.

• Connection event: triggered before the HTTP request response, and when the client establishes an underlying TCP connection with the server.
• Request event: after the TCP connection is established, the bottom layer of the HTTP module will abstract the HTTP request and response from the data stream. This event will be triggered when the HTTP request header is parsed.
• close event: triggered when the server.close() method is called to stop accepting new connections and all existing connections are disconnected.
• checkContinue event: when the client sends large data, it will not directly send the data, but first send a request with expect: 100 Continue in the header to the server, and the server will trigger the checkContinue event. If the server does not listen to this event, it will automatically respond to the status code of client 100 Continue, indicating that it accepts data upload. If it is not accepted, or when the client has more data, respond to 400 Bad Request to refuse the client to continue sending data.
• CONNECT event: triggered when the client initiates a CONNECT request. The connection request usually occurs during HTTP proxy. If you do not listen to this event, the connection initiating the request will be closed.
• Upgrade event: triggered when the client requests to upgrade the connection protocol.
• clientError event: when the connected client triggers the error event, this error will be passed to the server, and this event will be triggered.

(4) HTTP client

The HTTP module constructs the client by calling http.request(options, connect). Roughly the same as curl above:

var options = {
  hostname: '127.0.0.1',
  port: 1334,
  path: '/',
  method: 'GET'
}
var req = http.request(options, function (res) {
  console.log('STATUS: ' + res.statusCode)
  console.log('HEADERS: ' + JSON.stringify(res.headers))
  res.setEncoding('utf8')
  res.on('data', function (chunk) {
    console.log(chunk)
  })
})
req.end()

Execution:

$ node client.js
STATUS: 200
HEADERS: {"date":"Mon, 04 Jun 2018 15:34:30 GMT","connection":"keep-alive","transfer-encoding":"chunked"}
Hello World

The options are as follows:

• host
• hostname
• port: default 80
• localAddress: the local network card that establishes the network connection
• socketPath
• method: GET by default
• Path: request path. The default is/
• headers
• auth: Basic authentication. This value will be calculated as the Authorization part in the request header.

(5) HTTP proxy

The ClientRequest object provided by HTTP is also implemented based on the TCP layer. In the case of keepalive, an underlying session connection can be used for multiple requests. To reuse TCP connections, the HTTP module contains a default client proxy object, http.globalAgent.

http.globalAgent manages the connections created by each server (host + port). By default, each request can create up to 5 connections. Its essence is a connection pool.

When the HTTP client is called to initiate 10 HTTP requests to a server, only 5 requests are in concurrent status. Subsequent requests need to wait for a request to be completed before they are actually issued, which is the same as the browser's concurrency limit on the same domain name.

var agent = new http.Agent({
  maxSockets: 10
})
var options = {
  hostname: '127.0.0.1',
  port: 1334,
  path: '/',
  method: 'GET',
  agent: agent
}

You can also set the agent option to false to break away from connection pool management so that requests are not limited by concurrency.

(6) HTTP client events

• Response: triggered when the client receives a response from the server.
• socket: this event is triggered when the connection in the underlying connection pool is allocated to the current request object.
• CONNECT: when the client sends a CONNECT request to the server, if the server responds to the 200 status code, the client triggers.
• Upgrade: when the client initiates an upgrade request, if the server responds to the 101 Switching Protocols status, the client triggers.
• Continue: the client sends the expect: 100 Continue header message to the server, the service server responds to the 100 Continue status, and the client triggers it.