Recently, I encountered a process suspension problem when using Python's multiprocessing module. Let's record it here.

First, the minimum code of a multi process application is given.

import multiprocessing as mp

def produce(q):
    """producer"""
    for i in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
        q.put(i)
    print(f"Producer quit.")


def consume(q):
    """consumer"""
    while True:
        r = q.get()
        if r == 9:
            break
    print("Consumer quit.")


if __name__ == "__main__":
    q = mp.Queue()

    consumer = mp.Process(target=consume, args=(q,))
    consumer.start()

    producer = mp.Process(target=produce, args=(q,))
    producer.start()

    producer.join()
    consumer.join()

    print("Main quit.")

In this code, the main process builds a queue and then derives two sub processes: a producer and a consumer. The producer process fills the queue one by one with elements from a numeric list from 1 to 9. The consumer process takes out the elements in the queue one by one and exits when the last element is read. The output of this code run is as follows:

Producer quit.
Consumer quit.
Main quit.

In practice, for security reasons, the maximum capacity of the queue will be limited to avoid memory exhaustion. During queue initialization, you can specify the maximum size allowed through the maxsize parameter.

# There is no change in the function definition of producer and consumer, which is omitted here.

if __name__ == "__main__":
    q = mp.Queue(maxsize=3)  # Specifies the maximum number of elements that the queue is allowed to store

    consumer = mp.Process(target=consume, args=(q,))
    consumer.start()

    producer = mp.Process(target=produce, args=(q,))
    producer.start()

    producer.join()
    consumer.join()

    print("Main quit.")

At this time, run all the codes, and the output is as follows:

Producer quit.
Consumer quit.
Main quit.

After limiting the queue size, the application can still exit normally.

However, if you accidentally forget the size limit of the queue and inadvertently modify the code of the consumer process:

# No change in producer code, omitted.

def consume(q):
    """consumer"""
    while True:
        r = q.get()
        if r == 3:  # When 3 is read, the consumption queue is stopped.
            break
    print("Consumer quit.")
    
# The main process code is unchanged and omitted.

This modification will cause the consumer process to terminate and exit when it reads the number 3. At this time, run all the codes, and the output is as follows:

Consumer quit.

The consumer process has exited normally. However, the producer and the main process are suspended. The program is stuck.

In order to find the cause of the problem, try to print out the running process of the program. Print the currently read elements in the producer and consumer processes respectively.

def produce(q):
    """producer"""
    for i in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
        q.put(i)
        print(f"P: {i}")  # Print the elements stored in the queue
    print(f"Producer quit.")


def consume(q):
    """consumer"""
    while True:
        r = q.get()
        print(f"C: {r}")  # Print elements taken from the queue
        if r == 3:
            break
    print("Consumer quit.")

The program inputs are as follows:

P: 1
P: 2
P: 3
C: 1
P: 4
C: 2
C: 3
Consumer quit.
P: 5
P: 6

As you can see from the output, the consumer process exits after reading 3. The producer process continues to fill the queue with elements, the last time 6 is filled, and then there is no following. Since the last element taken out in the queue is 3, and the producer fills in from 1 to 9, there are three elements in the queue: 4, 5 and 6. Recall that the maximum size of the queue was set to 3. Obviously, the queue is full. At this point, the put method is waiting patiently in the queue space. Since the consumer process has exited, the queue will never have space. The process is stuck here.

There are different solutions to this problem.

The first is to avoid the consumer process exiting earlier than the producer process. As long as consumers don't quit, the queue will always usher in a free day.

The second is to set the queue filling operation to non blocking. Set the block parameter of the put method to False, so that if filling the queue fails, the program will report that the queue is full, and then continue to execute.

The third is to allow queue filling and blocking, but specify the maximum allowable waiting time. The default put method timeout of the queue is None, which means waiting until it is free. Set the timeout parameter for the put method. If the queue still has no space after waiting for a long enough time, an error that the queue is full will be reported. The program continues to run.

These three methods have their own advantages and disadvantages, and they need to be selected according to the actual situation. Ideally, the consumer should be treated as the last exit process through code logic. If the process behavior of consumers cannot be controlled due to special reasons, the latter two methods can be considered, and then try Except captures the error that the queue is full, and further combines retry to avoid data loss.