background
Although HTTP protocol has been widely used on the WEB, it is a stateless and one-way communication protocol. For some occasions that need to refresh page data in real time, it will be very embarrassing to implement it based on http: the front end needs to continuously initiate HTTP requests and connect to the server to obtain the latest data. This creates some problems:
- A large number of HTTP requests overflow, resulting in a waste of server resources;
- It also causes additional resource overhead for front-end scripts;
Therefore, we naturally think about whether we can establish a long TCP connection alone to realize two-way data communication between the front and back ends, so that the server can actively push the data to the front end. So websocket was born.
websocket
In fact, websocket borrows the HTTP protocol during the connection establishment phase, which is called "handshake". The designer is very wise to do so. Its advantage is that it can make use of some existing components of HTTP protocol (proxy, filter and authentication mechanism), reuse port 80 or 443 commonly used by HTTP, and so on. After establishing a connection based on HTTP, websocket can communicate freely.
Based on this, we generally divide the working process of websocket protocol into two parts: 1) handshake and 2) communication. There is also an end process, but it can generally be considered as part of the communication process. It will be parsed separately later.
Application scenario
- Web chat, instant messaging
- bullet chat
- Real time refresh of stock data
Case: wechat scanning QR code login on CSDN. The reason why the front-end page can actively jump after scanning is that the server actively pushes data to the front-end through websocket protocol.
websocket workflow
Opening Handshake
When the client sends a handshake to the server, the general format of the HTTP message is as follows:
GET /chat HTTP/1.1 Host: server.example.com Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Origin: http://example.com Sec-WebSocket-Protocol: chat, superchat Sec-WebSocket-Version: 13
After the server receives the request, its response message format is generally as follows:
HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= Sec-WebSocket-Protocol: chat
Among them, SEC websocket protocol refers to other user protocols supported by the client based on websocket. The server selects one of them in the response message. For example, different pages need different user protocols. We focus on the SEC websocket key field in the request message and the SEC websocket accept field in the response message. Literally, these two fields are mainly related to authentication. They help to confirm that the server can correctly support the websocket protocol, and also ensure that the request initiated by the client is a handshake request of websocket rather than an ordinary HTTP request.
In fact, SEC websocket key is a random string generated by the client. After receiving the SEC websocket key, the server will do two steps:
- 1 the websocket protocol defines a Globally Unique Identifier (GUID): 258EAFA5-E914-47DA-95CA-C5AB0DC85B11. The server splices the guid behind the SEC websocket key content, such as dGhlIHNhbXBsZSBub25jZQ==258EAFA5-E914-47DA-95CAC5AB0DC85B11;
- 2. Perform SHA-1 hash operation (160bit, 20 bytes) on the spliced string, and then perform base64 coding on the output result to obtain the content of SEC websocket accept we see in the response message s3plmbitxaq9kygzzhzrbk + Xoo =. Return it to the client.
After receiving the response message, the client performs the same calculation on the SEC websocket key, and then compares whether the results are consistent with those of the server. If they are consistent, the handshake process is completed, and the server proves that it is a qualified websocket server. Note that the header of the response message is HTTP/1.1 101 Switching Protocols.
After that, they put aside the HTTP protocol and happily entered the websocket communication link.
communication protocol
websocket communicates by sending a series of data frames. The frame format is shown in the following figure:
-
FIN: 1 bit
Indicates whether the current frame is the last frame of a complete piece of data, 0 indicates that there is data after it, and 1 indicates that this is the last frame; Note that FIN is in the highest position. -
RSV1, RSV2, RSV3: 1 bit each
Reserved field, must be 0 unless negotiated. -
opcode: 4 bits
Opcode indicating the type of payload data.* %x0 denotes a continuation frame
* %x1 denotes a text frame
* %x2 denotes a binary frame
* %x3-7 are reserved for further non-control frames
* %x8 denotes a connection close
* %x9 denotes a ping
* %xA denotes a pong
* %xB-F are reserved for further control frames -
MASK: 1 bit
Indicates whether the payload data is masked. If it is 1, then masking key is the mask. The protocol stipulates that the data sent by the client to the server should be masked. Strictly speaking, this mask processing certainly cannot play the role of encryption, but my understanding is that when others catch your packet, if there is no plaintext and only binary data, others may not even know that this is a websocket data frame. From this point of view, it is still meaningful. -
Payload len: 7 bits, 7+16 bits, or 7+64 bits
The length of payload is a very flexible design in websocket protocol. If the data length is less than 126 bytes (0 ~ 125), the length value is directly stored on the Payload len. There is no need for the later Extended payload length, which is directly masking key; If the data length is greater than 126 bytes, as required: 1) set the Payload len to 126, and the next two bytes are used to represent the data length; 2) If Payload len is set to 127, the next 8 bytes are used to represent the data length.
Please pay attention to the network byte order (big end byte order) adopted by websocket! When Payload len > = 126, remember to convert the received Payload len with ntohs, convert with htons before sending, and then send it to Payload len. -
Masking-key: 0 or 4 bytes
If MASK is set to 1, this is a 4-byte MASK, otherwise the data does not exist. The length of MASK data is not included in the length of payload. -
Payload Data
All the data. Extended data is also mentioned in the document, which is the data part negotiated by both parties and can be ignored.
In addition, we need to explain the working mode of the mask: XOR the data in the payload and the mask data. After the client XOR the data, the server uses the same mask to do XOR again to recover the data. Here is the use of the characteristics of XOR operation: A XOR B gets C, C and then XOR B can be restored to A. There are four mask bytes here. Each operation actually takes one of these four bytes in turn to XOR with the payload data. It will be more intuitive to see the code:
void umask(char *data,int len,char *mask)
{
int i;
for (i = 0;i < len;i ++)
*(data+i) ^= mask[i%4];
}
Closing Handshake
The ending process is much simpler than shaking hands. Both sides can initiate the end process. Just send a data frame with opcode 0x8, which is called the end frame. The end frame can also contain data, which can be used to prompt why it ends. For end frame data, websocket_ The rfc6455 document needs to point out that:
The Close frame MAY contain a body ...
... If there is a body, the first two bytes of the body MUST be a 2-byte unsigned integer (in network byte order) representing a status code
...
... the client SHOULD wait for the server to close the connection but MAY close the connection ... if it has not received a TCP Close from the server in a reasonable time period.
Roughly speaking, if the end frame is to be sent with payload, the first two bytes of data need to be an error code (stored in network byte order form), and the specific error code definition needs to see the original document. Secondly, after the client sends the end frame, it will wait for the server to close the connection. The client will close the connection first unless it times out.
When an end frame is sent at one end, no more data should be sent. After receiving the end frame, the other end sends the remaining data, and then sends a response of the end frame. After that, both sides can close the connection. If both ends initiate an end frame at the same time, both sides need to respond to the end frame before closing the connection.
Since TCP itself has waved four times, why does websocket have to make a separate end frame? websocket_ The rfc6455 document explains this:
The closing handshake is intended to complement the TCP closing handshake (FIN/ACK), on the basis that the TCP closing handshake is not always reliable end-to-end, especially in the presence of intercepting proxies and other intermediaries.
_
By sending a Close frame and waiting for a Close frame in response, certain cases are avoided where data may be unnecessarily lost. For instance, on some platforms, if a socket is closed with data in the receive queue, a RST packet is sent, which will then cause recv() to fail for the party that received the RST, even if there was data waiting to be read.
realization
This code is in the last blog epoll implementation of tcp server and Reactor model Add support for websocket protocol based on the code of.
Server state machine
Initial state - the socket has been established and the handshake has not been held
Handshake status - the client initiates a handshake message and responds correctly
Communication status - after the handshake process, it will always be in the communication status, and data can be sent and received according to the established protocol
End status - the client initiates the end frame, and the server actively closes the connection after returning the end frame
code implementation
Structure definition
/* Websocket Related definitions */
enum WS_STATUS // state
{
WS_INIT = 0,
WS_HANDSHAKE,
WS_DATATRANSFER,
WS_END
};
struct ws_opHeader
{
unsigned char opcode : 4,
RSV123 : 3,
FIN : 1;
unsigned char payloadLen : 7,
MASK : 1;
}__attribute__ ((packed)); // __ attribute__ ((packed)) is used to tell the compiler to cancel the optimal alignment of structures during compilation
struct ws_dataHeader126
{
unsigned short payloadLen;
}__attribute__ ((packed));
struct ws_dataHeader127
{
unsigned long long payloadLen;
}__attribute__ ((packed));
#define GUID ("258EAFA5-E914-47DA-95CA-C5AB0DC85B11")
/* Websocket Related definitions END */
#define MAX_BUFFER_SIZE 1024
struct sockitem
{
int sockfd;
int (*callback)(int fd, int events, void *arg);
int epfd;
char recvbuffer[MAX_BUFFER_SIZE]; // Receive buffer
char sendbuffer[MAX_BUFFER_SIZE]; // Send buffer
int recvlength; // Length of data in receive buffer
int sendlength; // Length of data in send buffer
int status; // Add status to implement the state machine
};
Among them__ attribute__ ((packed)) is used to tell the compiler to cancel the optimized alignment of structures during compilation and align them according to the actual number of bytes. This is an extension of GNU C to ANSI C.
auxiliary function
// It can be masked or unmasked
void umask(char *data,int len,char *mask)
{
int i;
for (i = 0;i < len;i ++)
*(data+i) ^= *(mask+(i%4));
}
int base64_encode(char *in_str, int in_len, char *out_str) {
BIO *b64, *bio;
BUF_MEM *bptr = NULL;
size_t size = 0;
if (in_str == NULL || out_str == NULL)
return -1;
b64 = BIO_new(BIO_f_base64());
bio = BIO_new(BIO_s_mem());
bio = BIO_push(b64, bio);
BIO_write(bio, in_str, in_len);
BIO_flush(bio);
BIO_get_mem_ptr(bio, &bptr);
memcpy(out_str, bptr->data, bptr->length);
out_str[bptr->length-1] = '\0';
size = bptr->length;
BIO_free_all(bio);
return size;
}
/* Read a row of data, excluding the end \ r\n */
int readline(char* allbuf,int level,char* linebuf) {
int len = strlen(allbuf);
for (;level < len; ++level) {
if(allbuf[level]=='\r' && allbuf[level+1]=='\n')
return level+2;
else
*(linebuf++) = allbuf[level];
}
return -1;
}
/* Remove spaces at the beginning and end of the string */
char* trim(char* str)
{
char* start;
if(str == NULL)
return NULL;
while(*str == ' ') str++;
start = str;
str += strlen(str) - 1;
while(*str == ' ')
{
*str = '\0';
str--;
}
return start;
}
/* 64 bit Local byte order to network byte order */
unsigned long long htonll(unsigned long long host)
{
return ((unsigned long long)htonl(host & 0xffffffff)) << 32 + htonl(host >> 32);
}
/* 64 bit Network byte order to local byte order */
unsigned long long ntohll(unsigned long long net)
{
return ((unsigned long long)ntohl(net & 0xffffffff)) << 32 + ntohl(net >> 32);
}
Core code
/* ws Handshake process processing */
int ws_handshake(struct sockitem *si)
{
int level = 0;
char linebuf[256];
char* keypos;
unsigned char sha1_data[SHA_DIGEST_LENGTH+1] = {0};
char base64_result[32]; // Final base64 results
printf("-------------------\nrequest\n%s\n", si->recvbuffer);
do
{
memset(linebuf, 0, 256);
level = readline(si->recvbuffer, level, linebuf);
/* Get sec websocket key */
if(keypos = strstr(linebuf, "Sec-WebSocket-Key"))
{
if(keypos = strstr(keypos, ":"))
{
keypos++; // Skip ':'
keypos = trim(keypos); // Eliminate leading and trailing spaces
printf("key: %s\n\n", keypos);
strcat(keypos, GUID); // Splice GUID
SHA1(keypos, strlen(keypos), sha1_data);
base64_encode(sha1_data, strlen(sha1_data), base64_result);
sprintf(si->sendbuffer, "HTTP/1.1 101 Switching Protocols\r\n" \
"Upgrade: websocket\r\n" \
"Connection: Upgrade\r\n" \
"Sec-WebSocket-Accept: %s\r\n" \
"\r\n", base64_result);
printf("response\n");
printf("%s\n", si->sendbuffer);
si->sendlength = strlen(si->sendbuffer);
si->callback = send_cb;
si->status = WS_DATATRANSFER; // The handshake is completed and enters the communication state
struct epoll_event ev;
ev.events = EPOLLOUT | EPOLLET;
ev.data.ptr = si;
epoll_ctl(si->epfd, EPOLL_CTL_MOD, si->sockfd, &ev);
break;
}
}
} while (level != -1);
return 0;
}
/* websocket Data frame unpacking */
char* ws_depack(char* frame, char* maskbuf, unsigned char* opcode)
{
unsigned long long payloadLen = 0;
char* payload;
char* mask;
if(frame == NULL || maskbuf == NULL)
return NULL;
struct ws_opHeader* op = (struct ws_opHeader*)frame; // Remove the relevant head
if(op->payloadLen == 126)
{
struct ws_dataHeader126* header126 = (struct ws_dataHeader126*)(frame + sizeof(struct ws_opHeader));
payloadLen = ntohs(header126->payloadLen);
mask = (char*)header126 + sizeof(struct ws_dataHeader126);
}
else if(op->payloadLen == 127)
{
struct ws_dataHeader127* header127 = (struct ws_dataHeader127*)(frame + sizeof(struct ws_opHeader));
payloadLen = ntohll(header127->payloadLen);
mask = (char*)header127 + sizeof(struct ws_dataHeader127);
}
else
{
payloadLen = op->payloadLen;
mask = frame + sizeof(struct ws_opHeader);
}
payload = mask;
if(op->MASK)
{
payload = mask + 4;
memcpy(maskbuf, mask, 4);
umask(payload, payloadLen, maskbuf);
}
*opcode = op->opcode;
return payload;
}
/* websocket Data frame packet. If maskbuf is NULL, it will not be masked */
int ws_enpack(char* frame, char* payload, unsigned long long payloadLen, char* maskbuf, char opcode)
{
char* mask;
if(frame == NULL)
return -1;
struct ws_opHeader op = {0};
op.FIN = 1; // All at once
op.opcode = opcode;
op.MASK = maskbuf ? 1 : 0;
if(payloadLen < 126)
{
op.payloadLen = payloadLen;
memcpy(frame, &op, sizeof(struct ws_opHeader));
mask = frame + sizeof(struct ws_opHeader);
}
else if(payloadLen < 0xffff)
{
op.payloadLen = 126;
struct ws_dataHeader126 header = {0};
header.payloadLen = htons(payloadLen);
memcpy(frame, &op, sizeof(struct ws_opHeader));
memcpy(frame + sizeof(struct ws_opHeader), &header, sizeof(struct ws_dataHeader126));
mask = frame + sizeof(struct ws_opHeader) + sizeof(struct ws_dataHeader126);
}
else
{
op.payloadLen = 127;
struct ws_dataHeader127 header = {0};
header.payloadLen = htonll(payloadLen);
memcpy(frame, &op, sizeof(struct ws_opHeader));
memcpy(frame + sizeof(struct ws_opHeader), &header, sizeof(struct ws_dataHeader126));
mask = frame + sizeof(struct ws_opHeader) + sizeof(struct ws_dataHeader127);
}
// Do you need to mask
if(maskbuf)
{
umask(payload, payloadLen, maskbuf);
memcpy(mask, maskbuf, 4);
mask += 4;
}
memcpy(mask, payload, payloadLen);
return mask - frame + payloadLen;
}
/* Communication process processing */
int ws_datatransfer(struct sockitem *si)
{
char* payload;
char mask[4];
unsigned long long payloadLen = 0;
unsigned char opcode;
// Unpack
payload = ws_depack(si->recvbuffer, mask, &opcode);
payloadLen = si->recvlength - (payload - si->recvbuffer);
printf("opcode:%d, recv len:%llu, payload: %s\n", opcode, payloadLen, payload);
if(opcode == 8) // If it is the end frame, there is no payload in the reply
payloadLen = 0;
// Packet
si->sendlength = ws_enpack(si->sendbuffer, payload, payloadLen, NULL, opcode);
memset(si->recvbuffer, 0, MAX_BUFFER_SIZE);
si->recvlength = 0;
si->callback = send_cb;
if(opcode == 8)
si->status = WS_END;
else
si->status = WS_DATATRANSFER;
struct epoll_event ev;
ev.events = EPOLLOUT | EPOLLET;
ev.data.ptr = si;
epoll_ctl(si->epfd, EPOLL_CTL_MOD, si->sockfd, &ev);
return 0;
}
int close_connection(struct sockitem *si, unsigned int event)
{
struct epoll_event ev;
close(si->sockfd);
ev.events = event;
ev.data.ptr = si;
epoll_ctl(si->epfd, EPOLL_CTL_DEL, si->sockfd, &ev);
free(si);
}
/* Write IO callback function */
int send_cb(int fd, int events, void *arg)
{
struct sockitem *si = arg;
struct epoll_event ev;
int clientfd = si->sockfd;
/* websocket */
int ret = send(clientfd, si->sendbuffer, si->sendlength, 0);
memset(si->sendbuffer, 0, MAX_BUFFER_SIZE);
si->sendlength = 0;
if(si->status == WS_END) // End status, close connection, cancel ev
close_connection(si, EPOLLOUT | EPOLLET);
else
{
si->callback = recv_cb;
ev.events = EPOLLIN;
ev.data.ptr = si;
epoll_ctl(si->epfd, EPOLL_CTL_MOD, si->sockfd, &ev);
}
return ret;
}
/* Read IO callback function */
int recv_cb(int fd, int events, void *arg)
{
struct sockitem *si = arg;
int clientfd = si->sockfd;
int ret = recv(clientfd, si->recvbuffer, MAX_BUFFER_SIZE, 0);
if(ret <= 0)
{
if(ret < 0)
{
if(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
{ // Interrupted direct return
return ret;
}
}
else
{
printf("# client disconnected...\n");
}
/* Delete the current client socket from epoll */
close_connection(si, EPOLLIN);
}
else
{
/* websocket status machine */
si->recvlength = ret;
switch(si->status)
{
case WS_HANDSHAKE:
ws_handshake(si);
break;
case WS_DATATRANSFER:
ws_datatransfer(si);
break;
case WS_END:
break;
default:
assert(0);
}
/* websocket END */
}
return ret;
}
test
You can use this website for testing: WebSocket online test
Supplement: web transmission ciphertext
(to be supplemented)