Introduction to producer consumer model

1. Why the producer consumer model is needed

Producer refers to the task of production data, and consumer refers to the task of consumption data.

When the producer's production capacity is far greater than the consumer's consumption capacity, the producer needs to wait for the consumer to consume before continuing to produce new data. Similarly, if the consumer's consumption capacity is far greater than the producer's production capacity, the consumer needs to wait for the producer to produce data before continuing to consume. This kind of waiting will cause low efficiency. In order to solve this problem, we will introduce The producer consumer model is proposed.

2. How to realize producer consumer model

The producer consumer model can be realized by introducing queue between processes. The concept of lock does not need to be considered by using queue, because the communication between processes is realized by queue;

Producer production data is written into the queue, and consumer consumption data is taken directly from the queue, thus decoupling between producer and consumer is realized.

Producers -- > queues < -- consumers

Queue implements producer consumer model

1. Consumer producer model code

from multiprocessing import Process, Queue
import time
# Consumer approach
def consumer(q, name):
while True:
res = q.get()
# if res is None: break
print("%s Ate %s" % (name, res))
# Producer method
def producer(q, name, food):
for i in range(3):
time.sleep(1) # Time delay of simulating watermelon production
res = "%s %s" % (food, i)
print("%s Produced %s" % (name, res))
# Put the produced vegetable in the queue
q.put(res)
if __name__ == "__main__":
#Create queue
q = Queue()
# Create producer
p1 = Process(target=producer, args=(q, "kelly", "watermelon"))
c1 = Process(target=consumer, args=(q, "peter",))
p1.start()
c1.start()
# p1.join()
# q.put(None)
print("Main process")

2. Execution results

2.1 results of directly executing the above code

A problem with direct execution is that the producer has finished production and does not send a stop signal to the consumer. Therefore, the consumer will always block q.get(), which makes the program unable to exit.

In order to solve the above problem and let the consumer exit automatically after consuming the producer's data, it is necessary to put an end signal into the queue when the producer process is introduced. When the consumer gets this signal, he or she exits the consumption process.

It is mainly modified in two places. By opening the comments in the code below, the consumer can exit the consumer process after receiving the end signal from the producer.

def consumer():
if res is None: break
if __name__ == "__main__":
p1.join()
q.put(None)

2.2 operation results after opening comments

After opening the comment, the consumer gets the end signal sent by the producer and can exit the program normally.

But if there are n consumers, they need to send N end signals. This way is not so simple, like the following code:

from multiprocessing import Process, Queue
import time
# Consumer approach
def consumer(q, name):
while True:
res = q.get()
if res is None: break
print("%s Ate %s" % (name, res))
# Producer method
def producer(q, name, food):
for i in range(3):
time.sleep(1) # Time delay of simulating watermelon production
res = "%s %s" % (food, i)
print("%s Produced %s" % (name, res))
# Put the produced vegetable in the queue
q.put(res)
if __name__ == "__main__":
# Create queue
q = Queue()
# Create producer
p1 = Process(target=producer, args=(q, "kelly", "watermelon"))
p2 = Process(target=producer, args=(q, "kelly2", "Banana"))
c1 = Process(target=consumer, args=(q, "peter",))
c2 = Process(target=consumer, args=(q, "peter2",))
c3 = Process(target=consumer, args=(q, "peter3",))
p1.start()
p2.start()
c1.start()
c2.start()
c3.start()
p1.join()
p2.join()
q.put(None)
q.put(None)
q.put(None)
print("Main process")

In fact, we now want to send an end signal to the queue after the producer has finished producing the data. python language provides another queue JoinableQueue([maxsize]) to solve this problem.

JoinableQueue implements producer consumer model

1. Introduction to JoinableQueue method

JoinableQueue([maxsize]) : A queue type which also supports join() and task_done() methods

q. Task? Done(): this method is used by the consumer to signal that the return item of q.get() has been processed.

q.join(): this method is called by the producer to block until all items in the queue are processed; the block will continue until each item in the queue calls the q.task_done() method.

2. JoinableQueue implementation producer consumer model source code

from multiprocessing import Process,JoinableQueue
import time
# Consumer approach
def consumer(q, name):
while True:
res = q.get()
if res is None: break
print("%s Ate %s" % (name, res))
q.task_done() # Send a signal to q.join(), indicating that a value has been taken from the queue and processed
# Producer method
def producer(q, name, food):
for i in range(3):
time.sleep(1) # Time delay of simulating watermelon production
res = "%s %s" % (food, i)
print("%s Produced %s" % (name, res))
# Put the produced vegetable in the queue
q.put(res)
q.join() # Wait for the consumer to get all the elements they put in the queue before they finish
if __name__ == "__main__":
# q = Queue()
q = JoinableQueue()
# Create producer
p1 = Process(target=producer, args=(q, "kelly", "watermelon"))
p2 = Process(target=producer, args=(q, "kelly2", "Blueberry"))
# Create consumer
c1 = Process(target=consumer, args=(q, "peter",))
c2 = Process(target=consumer, args=(q, "peter2",))
c3 = Process(target=consumer, args=(q, "peter3",))
c1.daemon = True
c2.daemon = True
c3.daemon = True
p_l = [p1, p2, c1, c2, c3]
for p in p_l:
p.start()
p1.join()
p2.join()
# 1. The main process waits for P1 and P2 to finish before execution
# 2. Because of the existence of q.join(), the producer will not end until the elements in the queue are consumed
# 3. When the producer is finished, it means that the consumer has finished consuming, and it can also be finished. Therefore, the consumer can be set as a daemons (exit with the exit of the main process)
print("Main process")

3. Operation results

It can be seen from the running results that the producer does not send the end signal to the consumer manually, but also implements the producer consumer model through JoinableQueue queue.