day21 network programming

Posted by aliento on Wed, 19 Jan 2022 10:45:01 +0100

day21 network programming (Part 2)

Course objective: learn the necessary knowledge of network programming development.

Today's summary:

  • OSI7 layer model
  • TCP and UDP
  • Sticky bag
  • Blocking and non blocking
  • IO multiplexing

1. OSI layer 7 model

The 7-layer OSI model may not be easy to understand, so let's explain it through a case:

Suppose you enter some keywords in the browser, find the corresponding IP through DNS, and then send the data. The internal will do the following:

  • Application layer: Specifies the format of data.

    "GET /s?wd=Hello HTTP/1.1\r\nHost:www.baidu.com\r\n\r\n"

  • Presentation layer: coding, compression (decompression), blocking, encryption (decryption) and other tasks of application layer data.

    "GET /s?wd=Hello HTTP/1.1\r\nHost:www.baidu.com\r\n\r\n Hello".encode('utf-8')

  • Session layer: responsible for establishing and interrupting the connection with the target.

    Before sending data, you need to send a "connection" request first, and then send data after establishing a connection with the remote. Of course, after sending, it also involves the operation of disconnecting.

  • Transport layer: establishing port to port communication actually determines the port information of both sides.

    Data:"GET /s?wd=Hello HTTP/1.1\r\nHost:www.baidu.com\r\n\r\n Hello".encode('utf-8')
Port:
	- Target: 80
	- Local: 6784

  • Network layer: mark target IP information (IP protocol layer)

    Data:"GET /s?wd=Hello HTTP/1.1\r\nHost:www.baidu.com\r\n\r\n Hello".encode('utf-8')
Port:
	- Target: 80
	- Local: 6784
IP: 
	- target IP: 110.242.68.3(Baidu)
	- local IP: 192.168.10.1

  • Data link layer: group data and set source and destination mac addresses

    Data:"POST /s?wd=Hello HTTP/1.1\r\nHost:www.baidu.com\r\n\r\n Hello".encode('utf-8')
Port:
	- Target: 80
	- Local: 6784
IP: 
	- target IP: 110.242.68.3(Baidu)
	- local IP: 192.168.10.1
MAC: 
	- target MAC: FF-FF-FF-FF-FF-FF 
	- Local machine MAC: 11-9d-d8-1a-dd-cd

  • Physical layer: transfer binary data on physical media.

    Send binary data through network cable

Each layer performs its own duties and ultimately ensures that the data is presented to users.

It can be simply understood as express delivery: 7 boxes are set outside the data. When the end user receives the box, he needs to open 7 boxes to get the data. During transportation, some boxes will be disassembled and replaced, for example:

Final shipping target: Shanghai ~ In Beijing (a transfer station may be required on the way), the box will be opened at the transfer station to view the information and forward it.
	- For secondary transfer station (layer-2 switch): open the box of the data link layer and view it mac Address information.
	- For level-3 transfer station (router or layer-3 switch): open the box of the network layer and view it IP Information.

In fact, it can only be reflected in the development process: application layer, presentation layer, session layer and transport layer. The processing of other layers is automatically completed in network equipment.

import socket

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('110.242.68.3', 80)) # Sent a packet to the server


key = "Hello"
# application layer
content = "GET /s?wd={} http1.1\r\nHost:www.baidu.com\r\n\r\n".format(key)
# Presentation layer
content = content.encode("utf-8")

client.sendall(content)
result = client.recv(8196)
print(result.decode('utf-8'))

# Session layer & transport layer
client.close()

2. UDP and TCP protocols

The protocol is actually some provisions for connecting and sending and receiving data.

In the OSI transport layer, in addition to defining port information, UDP or TCP protocols can also be specified. The details of connection and data transmission will be different according to different protocols.

  • UDP (User Data Protocol) user datagram protocol is a simple transport layer protocol for connecting to datagrams. UDP does not provide reliability. It just sends the datagrams transmitted by the application to the IP layer, but it does not guarantee that they can reach the destination. Because UDP does not establish a connection between the client and the server before transmitting datagrams, and there is no timeout retransmission mechanism, the transmission speed is very fast.

    Common are: voice call, video call, real-time game screen, etc.

  • TCP (Transmission Control Protocol) is a connection oriented protocol, that is, before sending and receiving data, you must establish a reliable connection with the other party, and then send and receive data.

    Common: website, mobile phone APP Data acquisition, etc.

2.1 UDP and TCP sample code

UDP examples are as follows:

UDP does not need to establish a connection.

  • Server

    import socket

server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)#  socket.SOCK_DGRAM UDP and TCP protocols, this parameter is different
server.bind(('127.0.0.1', 8002))

while True:
    data, (host, port) = server.recvfrom(1024) # block
    print(data, host, port)
    server.sendto("well".encode('utf-8'), (host, port))

  • client

    import socket

client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
while True:
    text = input("Please enter the content to send:")
    if text.upper() == 'Q':
        break
    client.sendto(text.encode('utf-8'), ('127.0.0.1', 8002))
    data, (host, port) = client.recvfrom(1024)
    print(data.decode('utf-8'))

client.close()

TCP examples are as follows:

  • Server

    import socket

# 1. Listen to the local IP and port
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8001))
sock.listen(5)

while True:
    # 2. Wait for someone to connect (blocking)
    conn, addr = sock.accept()

    # 3. Wait, the connector sends a message (blocking)
    client_data = conn.recv(1024)
    print(client_data)

    # 4. Reply to the connector
    conn.sendall(b"hello world")

    # 5. Close the connection
    conn.close()

# 6. Stop the server program
sock.close()

  • client

    import socket

# 1. Send a connection request to the specified IP
client = socket.socket()
client.connect(('127.0.0.1', 8001))

# 2. After the connection is successful, send a message. What does b mean here? [bytes like object]
# Error without b: TypeError: a bytes like object is required instead of 'str'
client.sendall(b'hello')

# 3. Wait for the reply of the message (blocking)
reply = client.recv(1024)
print(reply)

# 4. Close the connection
client.close()

2.2 TCP three handshakes and four waves

ACK:It is used to confirm the received data. The confirmed data is represented by the confirmation serial number.
SYN:Used for synchronization signal when establishing connection.
FIN:It means that there is no data to be sent later, which usually means that the established connection needs to be closed.


for the first time:
A client error occurred while establishing a connection syn package(seq=j) Go to the server and enter syn_sent(Half connected, synchronous (sent) status, waiting for server confirmation; syn: Synchronization sequence number.

The second time:
Server received syn Package, the client must be confirmed syn(ack=j+1),At the same time, it also happened syn package(seq=k)，Namely syn+ack Package, and the server enters syn_recv Status.(syn_recv It refers to that after the server is opened passively, it receives a message from the client syn And sent ack Status of the.)

third time:
Client received from server syn+ack Package and confirm the package to the server ack(ack=k+1),After the package occurs, the client and server enter ESTABLISHED(TCP The connection is successful) status, and three handshakes are completed. ESTABLISHED((formally established)

This is a common interview question.

What is? tcp Three handshakes?
Why do you shake hands three times when you connect and four times when you disconnect?

Why three handshakes?
It is mainly for information peer-to-peer and preventing dirty connections caused by request timeout.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If both parties in the network want to communicate based on TCP connection, they must go through:

  • To create a connection, the client and server should shake hands three times.

    # Server
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8001))
sock.listen(5)

while True:
    conn, addr = sock.accept() # Waiting for client connection
    ...

    # client
import socket
client = socket.socket()
client.connect(('127.0.0.1', 8001)) # Initiate connection

          client                                                Server

  1.  SYN-SENT    --> <seq=100><CTL=SYN>               --> SYN-RECEIVED

  2.  ESTABLISHED <-- <seq=300><ack=101><CTL=SYN,ACK>  <-- SYN-RECEIVED

  3.  ESTABLISHED --> <seq=101><ack=301><CTL=ACK>       --> ESTABLISHED

      
At this point, both the client and server have received an acknowledgment of the connection. The steps 1, 2 establish the connection parameter (sequence number) for one direction and it is acknowledged. The steps 2, 3 establish the connection parameter (sequence number) for the other direction and it is acknowledged. With these, a full-duplex communication is established.

  • Transmit data

    In the process of sending and receiving data, there will be a response only when there is data transmission( ack)，without ack，Then the internal will try to send repeatedly.

  • Close the connection, and the client and server should wave four times.

    import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8001))
sock.listen(5)
while True:
    conn, addr = sock.accept()
	...
    conn.close() # Close connection
sock.close()

    import socket

client = socket.socket()
client.connect(('127.0.0.1', 8001))
...
client.close() # Close connection

           TCP A                                                TCP B

  1.  FIN-WAIT-1  --> <seq=100><ack=300><CTL=FIN,ACK>  --> CLOSE-WAIT

  2.  FIN-WAIT-2  <-- <seq=300><ack=101><CTL=ACK>      <-- CLOSE-WAIT

  3.  TIME-WAIT   <-- <seq=300><ack=101><CTL=FIN,ACK>  <-- LAST-ACK

  4.  TIME-WAIT   --> <seq=101><ack=301><CTL=ACK>      --> CLOSED

3. Sticking package

When two computers send and receive data, they do not directly transmit the data to each other.

  • For the sender, when sending a message by sendall/send, the data is first sent to the write buffer of his own network card, and then the buffer sends the data to the read buffer of the other network card.
  • For the receiver, when recv receives a message, it obtains data from the read buffer of its own network card.

Therefore, if the sender sends two pieces of information continuously and quickly, the receiver will consider it as one piece of information when reading, that is, the two packets are stuck together. For example:

# socket client (sender)
import socket

client = socket.socket()
client.connect(('127.0.0.1', 8001))
# You can also use send, which is not recommended. When the buffer location is not enough, we can only write part of the data to the buffer, but we can't directly perceive it. We can also know it by the return value.
client.sendall('Emma Playing'.encode('utf-8'))
client.sendall('game'.encode('utf-8'))

client.close()


# socket server (receiver)
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8001))
sock.listen(5)
conn, addr = sock.accept()

client_data = conn.recv(1024)
print(client_data.decode('utf-8'))

conn.close()
sock.close()

# Output: Emma is playing a game. [two messages were actually sent, but the receiver did receive one message.]

How to solve the problem of sticking package?

Each time a message is sent, the message is divided into two parts: header (fixed byte length) and data. For example, in the header, the length of the following data is represented by 4 bytes.

  • To send data, first send the length of the data, and then send the data (or splice it together and then send it).
  • When receiving data, you can know the data length in your packet by reading 4 bytes first, and then read the data according to the length.

The header needs a number and is fixed to 4 bytes. This function can be realized with the help of python's struct package:

import struct

# ########### The value is converted to a fixed range of 4 bytes -2147483648 <= number <= 2147483647  ###########
v1 = struct.pack('i', 199)
print(v1)  # b'\xc7\x00\x00\x00'

for item in v1:
    print(item, bin(item))
    
# ########### 4 Convert bytes to numbers ###########
v2 = struct.unpack('i', v1) # v1= b'\xc7\x00\x00\x00'
print(v2) # (199,)

Example code:

  • Server

    import socket
import struct
# AF_INET Internet domain SOCK_STREAM socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
sock.bind(('127.0.0.1', 8001))
sock.listen(5)
conn, addr = sock.accept()

# Fixed read 4 bytes
header1 = conn.recv(4)
data_length1 = struct.unpack('i', header1)[0] # Get data byte length
has_recv_len = 0
data1 = b"" # b byte
while True:
    length = data_length1 - has_recv_len
    if length > 1024:
        lth = 1024
    else:
        lth = length
    chunk = conn.recv(lth) # If the transmitted data is very long and may not be received at one time, you can calculate the length and use recv again until it is received. 1024 * 8 = 8192
    data1 += chunk
    has_recv_len += len(chunk)
    if has_recv_len == data_length1:
        break
print(data1.decode('utf-8'))

# Fixed read 4 bytes
header2 = conn.recv(4)
data_length2 = struct.unpack('i', header2)[0] # Data byte length
data2 = conn.recv(data_length2) # length
print(data2.decode('utf-8'))

conn.close()
sock.close()

  • client

    import socket
import struct
                        
client = socket.socket()
client.connect(('127.0.0.1', 8001))
                        
# Article 1 data
data1 = 'emma Playing'.encode('utf-8')
                        
header1 = struct.pack('i', len(data1))
                        
client.sendall(header1)
client.sendall(data1)
                        
# The second data, first send its length, and then send the data
data2 = 'game'.encode('utf-8')
header2 = struct.pack('i', len(data2))
client.sendall(header2)
client.sendall(data2)
                        
client.close()

Case: Message & file upload

  • Server

    import os
import json
import socket
import struct


def recv_data(conn, chunk_size=1024):
    # Get header information: data length
    has_read_size = 0
    bytes_list = []  # Byte list
    while has_read_size < 4:
        chunk = conn.recv(4 - has_read_size)
        has_read_size += len(chunk)
        bytes_list.append(chunk)
    header = b"".join(bytes_list)
    data_length = struct.unpack('i', header)[0] # Replace with integer

    # get data
    data_list = []
    has_read_data_size = 0
    while has_read_data_size < data_length:
        size = chunk_size if (data_length - has_read_data_size) > chunk_size else data_length - has_read_data_size
        chunk = conn.recv(size)
        data_list.append(chunk)
        has_read_data_size += len(chunk)

    data = b"".join(data_list)

    return data


def recv_file(conn, save_file_name, chunk_size=1024):
    save_file_path = os.path.join('files', save_file_name)
    # Get header information: data length, read fixed four bytes, and use unpack to convert to integer
    has_read_size = 0
    bytes_list = []

    while has_read_size < 4:
        chunk = conn.recv(4 - has_read_size) #  Subtract the data that has been read
        bytes_list.append(chunk)
        has_read_size += len(chunk)
    header = b"".join(bytes_list) # join string splicing, splicing empty bytes and bytes in the byte list???
    data_length = struct.unpack('i', header)[0]


    # get data
    file_object = open(save_file_path, mode='wb')
    has_read_data_size = 0
    while has_read_data_size < data_length:
        #
        size = chunk_size if (data_length - has_read_data_size) > chunk_size else data_length - has_read_data_size
        chunk = conn.recv(size)
        file_object.write(chunk)
        file_object.flush()
        has_read_data_size += len(chunk)
    file_object.close()


def run():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    # IP can be reused. When the program exits abnormally, it may occupy IP. Using this code, IP can be reused multiple times
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    sock.bind(('127.0.0.1', 8001))
    sock.listen(5)
    while True:
        conn, addr = sock.accept()

        while True:
            # Get message type
            message_type = recv_data(conn).decode('utf-8')
            if message_type == 'close':  # Four waves, empty content.
                print("Close connection")
                break
            # File: {'msg_type':'file', 'file_name':"xxxx.xx"}
            # Message: {'msg_type':'msg'}
            message_type_info = json.loads(message_type)
            if message_type_info['msg_type'] == 'msg':
                data = recv_data(conn)
                print("Message received:", data.decode('utf-8'))
            else:
                file_name = message_type_info['file_name']
                print("File received, to save to:", file_name)
                recv_file(conn, file_name)

        conn.close()
    sock.close()



if __name__ == '__main__':
    run()

  • client

    import os
import json
import socket
import struct

# Define functions and send text messages
def send_data(conn, content): #Receive a connection: conn. check that the place where the call is made is the connection object passed through the client
    data = content.encode('utf-8') # Convert to byte type
    header = struct.pack('i', len(data)) # Get its length
    conn.sendall(header) # Length of data sent
    conn.sendall(data) # send data

# Define functions and upload files
def send_file(conn, file_path):
    file_size = os.stat(file_path).st_size # Gets the size of the file
    header = struct.pack('i', file_size)
    conn.sendall(header) # Size of data sent

    has_send_size = 0
    file_object = open(file_path, mode='rb')
    while has_send_size < file_size:
        chunk = file_object.read(2048) # 2048 bytes read at a time
        conn.sendall(chunk) #  Content of data sent
        has_send_size += len(chunk) # The loop ends when the read data is equal to the file size
    file_object.close()


def run():
    # Establish connection
    client = socket.socket()
    client.connect(('127.0.0.1', 8001))

    while True:
        """
        Please send a message in the format:
            - Message: msg|How do you do
            - File: file|xxxx.png
        """
        content = input(">>>")  #Wait for user input, msg or file
        if content.upper() == 'Q':
            send_data(client, "close") # The fourth wave, the content sent is empty
            break
        input_text_list = content.split('|')
        if len(input_text_list) != 2: # Separated by "|", if it is not equal to 2, the format error will be prompted
            print("Format error, please re-enter")
            continue

        message_type, info = input_text_list

        # Send a message
        if message_type == 'msg':

            # Message type
            send_data(client, json.dumps({"msg_type": "msg"}))

            # Send content
            send_data(client, info)

        # Send documents
        else:
            file_name = info.rsplit(os.sep, maxsplit=1)[-1] # os.sep parameter, automatically distinguishing system separator and / or\

            # Message type
            send_data(client, json.dumps({"msg_type": "file", 'file_name': file_name}))

            # Send content
            send_file(client, info)

    client.close()


if __name__ == '__main__':
    run()

4. Blocking and non blocking

By default, the network programming codes we write are blocked (waiting). Blocking is mainly reflected in:

# ################### socket Server (receiver)###################
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8001))
sock.listen(5)

# block
conn, addr = sock.accept()

# block
client_data = conn.recv(1024)
print(client_data.decode('utf-8'))

conn.close()
sock.close()


# ################### socket Client (sender) ###################
import socket

client = socket.socket()

# block
client.connect(('127.0.0.1', 8001))

client.sendall('emma Playing games'.encode('utf-8'))

client.close()

If you want to make the code non blocking, you need to write this:

# ################### socket Server (receiver)###################
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.setblocking(False) # Plus, it becomes non blocking, and True is blocking

sock.bind(('127.0.0.1', 8001))
sock.listen(5)

# Non blocking, BlockingIOError, wants to receive a client connection
conn, addr = sock.accept()

# Non blocking,
client_data = conn.recv(1024)
print(client_data.decode('utf-8'))

conn.close()
sock.close()

# ################### socket Client (sender) ###################
import socket

client = socket.socket()

client.setblocking(False) # Plus, it becomes non blocking

# Non blocking
client.connect(('127.0.0.1', 8001))

client.sendall('emma Playing games'.encode('utf-8'))

client.close()

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-i6ijcyjz-164299494969) (images / image-202112211408039. PNG)]

If the code becomes non blocking, the BlockingIOError exception will be thrown when the program encounters accept, recv and connect.

This is not an error in code writing, but a fixed error thrown because no relevant IO request is received after the original IO blocking becomes non blocking.

Non blocking code is generally combined with IO multiplexing, which can play a greater role.

5. IO multiplexing

I/O multiplexing means that multiple descriptors can be monitored through a mechanism. Once a descriptor is ready (generally read ready or write ready), it can notify the program to perform corresponding read-write operations.

Note: "a descriptor is usually a number,Represents an open file,process,disk inode wait. The kernel has a large array to maintain this thing. It can also be understood that the descriptor is a in the database table id field,The resource corresponding to the descriptor is the field itself. But the database here is the kernel. "

IO multiplexing + non blocking enables the TCP server to process the requests of multiple clients at the same time, for example:

# ################### socket Server ###################
import select
import socket

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(False)  # Plus, it becomes non blocking
server.bind(('127.0.0.1', 8001))
server.listen(5)

inputs = [server, ] # socket object list - > [server, first client connection Conn, second client connection conn]

while True:
    # When the socket object in the parameter 1 sequence is readable (accetp and read), the changed object is obtained and added to the r list.
    # r = [] when there is no new connection received, r is an empty list, and the code in the loop will not be executed at this time
    # R = [server,] when there is a new connection for the first time, the server changes. At this time, r = [server,]
    # r = [first client connection conn,]
    # r = [server,]
    # r = [first client connection Conn, second client connection conn]
    # r = [Second client connection conn,] #The second client is connected to conn. if there is no data, it will no longer listen and will be removed from the list again
    # Who has changed is in this list. r = [changed connection,]
    r, w, e = select.select(inputs, [], [], 0.05)
    for sock in r:
        # server
        if sock == server:
            conn, addr = sock.accept() # Receive new connections.
            print("New connection")
            # Conn.sendall() can send a message to the customer service terminal through conn.sendall()
            # conn.recv("xx") receives the message
            inputs.append(conn)
        else: # When the connection is received, sock= The server will execute the code under else
            data = sock.recv(1024)
            if data: # If the data is empty, close the connection and remove it from the list
                print("Message received:", data)
            else:
                print("Close connection")
                inputs.remove(sock)
	# Deal with other matters for 20s
"""
advantage:
	1. When no new links come, you can handle other things
	2. Let the server support multiple clients to connect at the same time.
"""

select Module parameter interpretation:
select Is for many file descriptors(abbreviation fd)For monitoring, it has three parameters:

rlist -- wait until ready for reading

wlist -- wait until ready for writing

xlist -- wait for an "exceptional condition"

The first parameter monitors the incoming data fd List, select Monitor this list and wait for these fd Send data, once the data is sent(It's ready to read)，It returns a readable fd list

The second parameter is the of the monitored data fd List, select Monitor this list and wait for these fd Send out data once fd Ready to send(It's ready to write)，Returns a writable fd list

Third parameter monitoring fd List and return the exception fd list

The fourth parameter is the detection time period

# ################### socket Client 1 ###################
import socket

client = socket.socket()
# block
client.connect(('127.0.0.1', 8001))

while True:
    content = input(">>>")
    if content.upper() == 'Q':
        break
    client.sendall(content.encode('utf-8'))

client.close()

# ################### socket Client 2 ###################
import socket

client = socket.socket()
# block
client.connect(('127.0.0.1', 8001))


while True:
    content = input(">>>")
    if content.upper() == 'Q':
        break
    client.sendall(content.encode('utf-8'))

client.close() # If you disconnect from the server (wave four times), you will want to send empty data to the server by default.

IO multiplexing + non blocking enables TCP clients to send multiple requests at the same time, such as sending a request to download pictures to a website.

import socket
import select
import uuid
import os

client_list = []  # socket object list

# Create 5 socket s
for i in range(5):
    client = socket.socket()
    client.setblocking(False)

    try:
        # Connect to Baidu. Although there is an abnormal BlockingIOError, the connection request is sent to Baidu normally
        client.connect(('47.98.134.86', 80))
    except BlockingIOError as e:
        pass

    client_list.append(client)

recv_list = []  # Put the socket that has successfully connected and sent the request to download the picture
while True:
    # w = [first socket object,]
    # r = [socket object,]
    r, w, e = select.select(recv_list, client_list, [], 0.1)
    for sock in w:
        # Connect successfully, send data
        # Request to download pictures
        sock.sendall(b"GET /nginx-logo.png HTTP/1.1\r\nHost:47.98.134.86\r\n\r\n")
        recv_list.append(sock)
        client_list.remove(sock)

    for sock in r:
        # After the data is sent successfully, the received return value (picture) is written to the local file
        data = sock.recv(8196)
        content = data.split(b'\r\n\r\n')[-1]
        random_file_name = "{}.png".format(str(uuid.uuid4()))
        with open(os.path.join("images", random_file_name), mode='wb') as f:
            f.write(content)
        recv_list.remove(sock)

    if not recv_list and not client_list:
        break
        
"""
advantage:
	1. Concurrency can be forged.
"""

Based on the characteristics of IO multiplexing + non blocking, no matter the server or client writing the socket, the performance can be improved. among

  • IO multiplexing to monitor whether the socket object has changed (whether the connection is successful, whether there is data coming, etc.).
  • Non blocking, socket connect and recv processes no longer wait.

Note: IO multiplexing can only be used to monitor whether IO objects have changed. Common include: whether files are readable and writable, input and output of computer terminal devices, and network requests (common).

In Linux operating systematization, IO multiplexing has three modes: select, poll and epoll. (windows only supports select mode)

Monitor whether the socket object has a new connection or new data.

select
 
select It first appeared in April 1983.2BSD In, it passes through a select()The system call is used to monitor the array of multiple file descriptors when select()After returning, the ready file descriptors in the array will be modified by the kernel, so that the process can obtain these file descriptors for subsequent read and write operations.
select At present, it is supported on almost all platforms, and its good cross platform support is also one of its advantages. In fact, from now on, this is also one of its few remaining advantages.
select One disadvantage of is that there is a maximum limit on the number of file descriptors that a single process can monitor Linux It is generally 1024, but this limit can be raised by modifying the macro definition or even recompiling the kernel.
In addition, select()The maintained data structure storing a large number of file descriptors increases linearly with the increase of the number of file descriptors. At the same time, due to the delay of network response time, a large number of TCP The connection is inactive, but the select()For all socket A linear scan is performed, so it also wastes some overhead.
 
poll
 
poll Born in 1986 System V Release 3，It and select It doesn't make much difference in essence, but poll There is no limit to the maximum number of file descriptors.
poll and select There is also a disadvantage that the array containing a large number of file descriptors is copied between the user state and the address space of the kernel as a whole. Regardless of whether these file descriptors are ready or not, its overhead increases linearly with the increase of the number of file descriptors.
In addition, select()and poll()After the ready file descriptor is told to the process, if it is not modified by the process IO Operation, then the next call select()and poll()These file descriptors will be reported again, so they generally do not lose the ready message. This method is called horizontal trigger( Level Triggered). 
 
#### The above two methods are inefficient

epoll
 
until Linux2.6 There is an implementation method directly supported by the kernel, that is epoll，It has almost all the advantages mentioned before and is recognized as Linux2.6 Multi channel with the best performance I/O Ready notification method.
epoll It can support both horizontal trigger and edge trigger( Edge Triggered，Only tell the process which file descriptors have just become ready. It only says it once. If we do not take action, it will not tell again. This method is called edge trigger). Theoretically, the performance of edge trigger is higher, but the code implementation is quite complex.
epoll Similarly, only those ready file descriptors are notified, and when we call epoll_wait()When you get the ready file descriptor, what is returned is not the actual descriptor, but a value representing the number of ready descriptors. You only need to epoll You can get the corresponding number of file descriptors from the specified array in turn. Memory mapping is also used here( mmap)Technology, which completely eliminates the overhead of copying these file descriptors during system call.
Another essential improvement is epoll Event based ready notification is adopted. stay select/poll In, the kernel scans all monitored file descriptors only after the process calls a certain method epoll Prior adoption epoll_ctl()To register a file descriptor. Once a file descriptor is ready, the kernel will adopt a similar method callback The callback mechanism quickly activates this file descriptor when the process calls epoll_wait()You'll be notified when.

#### The callback mechanism is equivalent to that whoever has data raises his hand and tells me, I will go to him to get data, which is more efficient.

Supplement: socket + non blocking + IO multiplexing (as long as IO operation objects and files can be monitored).

summary

  1. OSI layer 7 model

    Application layer, presentation layer, session layer, transport layer, network layer, data link layer and physical layer.

  2. Difference between UDP and TCP

    UDP，Fast, but can not guarantee the accuracy of data.
TCP，You need to create a reliable connection before sending and receiving data( ack).

  3. TCP's three handshakes and four waves

  4. Why are there sticky bags? How to solve it?

  5. How to make socket requests non blocking?

  6. What is the role of IO multiplexing?

    Monitor multiple IO Whether the object has changed (readable)/Writable).
    • IO multiplexing + non blocking + socket server allows the server to process the requests of multiple clients at the same time.
    • The IO multiplexing + non blocking + socket client can initiate multiple requests to the server at the same time.

Topics: Python network http udp

Hot Topics

©2024 Programmer Think