1. Basic concepts
AMQP protocol: Advanced message queuing protocol, a network protocol for asynchronous message transmission between processes. rabbitmq is developed based on AMQP protocol
General workflow: Publisher - > exchange - > queue - > consumer
Broker: agent, consisting of Exchange and Queue The process of connecting producers and consumers to realize the message Queue and routing function in AMPQ protocol
Virtual Host: Virtual Host. There can be multiple exchanges and queues in a Virtual Host for permission control
Exchange: the switch receives messages sent by the producer and according to routing_key routes the message to the specified Queue
Queue: message queue, which stores messages to be consumed It consists of headers and body. Headers contain various attribute parameters of the message added by the producer. Body is the real data content sent
Binding: through routing_key binds Exchange to Queue routing_key cannot be used as an arbitrary string, usually with "." Split, such as "register.shanghai", "register.beijing", "register. #" regular * (asterisk) represents any word# (hash) represents 0 or more words
Channel: channel, the communication channel between consumers and brokers, and the virtual connection established on the TCP connection Many channels can be established on a TCP connection to reduce system overhead and improve performance
2. Working mode
generalization:
1. Simple mode: the simplest one-to-one
2. Work queue mode: one to many A producer corresponds to multiple consumers, but each message can only be consumed by one of them
Polling distribution: send messages to each consumer in turn, and one consumer will send the next after processing Example: a consumption No. 1, 4, 7 Message, B consumption 2,5,8 Message, C consumption 3,6,9 A message
Fair distribution: send pending messages to consumers as long as they have free time Improve efficiency relative to polling distribution
3. Publish / subscribe mode: messages generated by one producer can be consumed by multiple consumers at the same time The producer sends the message to the broker, and the Exchange forwards the message to each Queue bound to the switch. The consumer listens to his own Queue for consumption
4. Routing mode: the producer sends the message to the broker, and the Exchange sends the message according to the routing_ The key is distributed to different queues, and the consumer also distributes the key according to the routing_key find the corresponding Queue for consumption
5. Topic mode: Based on the routing mode, routing_key supports regular matching
6. RPC mode: the RPC function is realized through the message queue. The client sends the message to the consumption queue, and the server consumption message execution program returns the result to the client, that is, sends the result to the callback queue
Summary:
Simple mode and work queue mode are of one kind: they can be used without declaring the switch, let alone specifying the switch type
Publish / subscribe mode, routing mode and topic mode are of one kind: the switch needs to be declared and the switch type needs to be specified
RPC mode is a kind of two - way production and consumption mode
3. python demo
producer
import pika
import json
import datetime
import time
import random
class TestPublisher(object):
def __init__(self):
self.username = 'sanford'
self.password = '123456'
self.host = 'localhost'
self.port = 5672
self.virtual_host = 'sanford_host'
def publisher_00(self, msg_type, data):
"""
Simple mode/Work queue mode
"""
# Login credentials at connection creation
credentials = pika.PlainCredentials(username=self.username, password=self.password)
# Parameter setting
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
# Create a blocking connection
connection = pika.BlockingConnection(params)
# Create channel
channel = connection.channel()
# Declare queue persistence: durable=True. The queue still exists after the service is restarted
channel.queue_declare(queue='queue_00', durable=True)
# Message property setting message persistence: delivery_mode=2
properties = pika.BasicProperties(headers={'msg-type': msg_type},
delivery_mode=2)
# Routing when exchange is the default ''_ Key is queue
channel.basic_publish(exchange='',
routing_key='queue_00',
body=json.dumps(data),
properties=properties)
connection.close()
def publisher_01(self, msg_type, data):
"""
release/subscription model (fanout)
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_01', exchange_type='fanout', durable=True)
properties = pika.BasicProperties(headers={'msg-type': msg_type},
delivery_mode=2)
# This mode does not require routing_key can be set to ''
channel.basic_publish(exchange='exchange_01',
routing_key='',
body=json.dumps(data),
properties=properties)
connection.close()
def publisher_02(self, msg_type, data):
"""
Routing mode(direct)
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_02', exchange_type='direct', durable=True)
properties = pika.BasicProperties(headers={'msg-type': msg_type},
delivery_mode=2)
# Specify routing_key
channel.basic_publish(exchange='exchange_02',
routing_key='routing_key_02',
body=json.dumps(data),
properties=properties)
connection.close()
def publisher_03(self, msg_type, data):
"""
Theme mode(topic),Implement distribution,
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_03', exchange_type='topic', durable=True)
properties = pika.BasicProperties(headers={'msg-type': msg_type},
delivery_mode=2)
# Splicing routing_key
channel.basic_publish(exchange='exchange_03',
routing_key='routing_key.{0}'.format(msg_type),
body=json.dumps(data),
properties=properties)
connection.close()
if __name__ == '__main__':
test_publisher = TestPublisher()
# Simple mode / work queue mode
# msg_type_00 = 'msg_type_00'
# for i in range(20):
# time.sleep(1)
# data_00 = {'id': i, 'name': 'jay', 'send_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
# test_publisher.publisher_00(msg_type_00, data_00)
# Publish / subscribe mode (fanout)
# msg_type_01 = 'msg_type_01'
# for i in range(20):
# time.sleep(1)
# data_01 = {'id': i, 'name': 'jay', 'send_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
# test_publisher.publisher_01(msg_type_01, data_01)
# Routing mode (direct)
# msg_type_02 = 'msg_type_02'
# for i in range(20):
# time.sleep(1)
# data_02 = {'id': i, 'name': 'jay', 'send_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
# test_publisher.publisher_02(msg_type_02, data_02)
# Topic mode to realize distribution
msg_type_list = ['msg_type_03.0', 'msg_type_03.1', 'msg_type_03.2']
for i in range(20):
msg_type_03 = random.choice(msg_type_list)
time.sleep(1)
data_03 = {'id': i, 'name': 'jay', 'send_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
test_publisher.publisher_03(msg_type_03, data_03)
consumer
import pika
import json
import time
import random
import datetime
import sys
class TestConsumer(object):
def __init__(self):
self.username = 'sanford'
self.password = '123456'
self.host = 'localhost'
self.port = 5672
self.virtual_host = 'sanford_host'
@staticmethod
def callback_00(channel, method, properties, body):
"""
Callback function
"""
time.sleep(random.randint(1, 10))
headers = properties.headers
data = json.loads(body)
data['end_time'] = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(headers, data)
# Manually confirm that the consumption has been successful when auto_ When ack = false
channel.basic_ack(delivery_tag=method.delivery_tag)
def consumer_00(self):
"""
Simple mode/Work queue mode
"""
# Login credentials at connection creation
credentials = pika.PlainCredentials(username=self.username, password=self.password)
# Parameter setting
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
# Create a blocking connection
connection = pika.BlockingConnection(params)
# Create channel
channel = connection.channel()
# Declare queue persistence: durable=True. The queue still exists after the service is restarted
channel.queue_declare(queue='queue_00', durable=True)
# Fair distribution (polling distribution without this line) prefetch_count=1 if a message among consumers is not processed, it will not continue to send messages to this consumer
channel.basic_qos(prefetch_count=1)
# auto_ack=True automatically confirms that the consumption has been successful
channel.basic_consume(queue='queue_00',
on_message_callback=self.callback_00)
channel.start_consuming()
def consumer_01(self):
"""
release/subscription model (fanout)
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_01', exchange_type='fanout', durable=True)
# Declare the queue. When the queue is an empty string, a unique queue name will be created exclusive=True, only the current connection is allowed to access
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
# The queue is bound to the switch through the routing key. This mode does not require routing_ The default value for key consumption is None
channel.queue_bind(exchange='exchange_01', queue=queue_name)
channel.basic_consume(queue=queue_name,
on_message_callback=self.callback_00)
channel.start_consuming()
def consumer_02(self):
"""
Routing mode(direct)
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_02', exchange_type='direct', durable=True)
# Declare the queue. When the queue is an empty string, a unique queue name will be created exclusive=True, only the current connection is allowed to access
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
# The queue is bound to the switch through the routing key. This mode requires routing_key for consumption
channel.queue_bind(exchange='exchange_02', queue=queue_name, routing_key='routing_key_02')
channel.basic_consume(queue=queue_name,
on_message_callback=self.callback_00)
channel.start_consuming()
def consumer_03_0(self):
"""
Theme mode(topic),Implement distribution routing_key.msg_type_03.0
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_03', exchange_type='topic', durable=True)
# Declare the queue. When the queue is an empty string, a unique queue name will be created exclusive=True, only the current connection is allowed to access
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
# The queue is bound to the switch through the routing key. This mode requires routing_key for consumption
channel.queue_bind(exchange='exchange_03', queue=queue_name, routing_key='routing_key.msg_type_03.0')
channel.basic_consume(queue=queue_name,
on_message_callback=self.callback_00)
channel.start_consuming()
def consumer_03_1(self):
"""
Theme mode(topic),Implement distribution routing_key.msg_type_03.1
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_03', exchange_type='topic', durable=True)
# Declare the queue. When the queue is an empty string, a unique queue name will be created exclusive=True, only the current connection is allowed to access
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
# The queue is bound to the switch through the routing key. This mode requires routing_key for consumption
channel.queue_bind(exchange='exchange_03', queue=queue_name, routing_key='routing_key.msg_type_03.1')
channel.basic_consume(queue=queue_name,
on_message_callback=self.callback_00)
channel.start_consuming()
def consumer_03_2(self):
"""
Theme mode(topic),Implement distribution routing_key.msg_type_03.#
* (asterisk) Represents any word
# (hash) 0 or more words
"""
credentials = pika.PlainCredentials(username=self.username, password=self.password)
params = pika.ConnectionParameters(host=self.host,
port=self.port,
virtual_host=self.virtual_host,
credentials=credentials)
connection = pika.BlockingConnection(params)
channel = connection.channel()
# Declare the switch persistence of the specified type of switch: durable=True. The switch still exists after the service is restarted
channel.exchange_declare(exchange='exchange_03', exchange_type='topic', durable=True)
# Declare the queue. When the queue is an empty string, a unique queue name will be created exclusive=True, only the current connection is allowed to access
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
# The queue is bound to the switch through the routing key. This mode requires routing_key for consumption
channel.queue_bind(exchange='exchange_03', queue=queue_name, routing_key='routing_key.msg_type_03.#')
channel.basic_consume(queue=queue_name,
on_message_callback=self.callback_00)
channel.start_consuming()
if __name__ == '__main__':
test_consumer = TestConsumer()
# Simple mode / work queue mode
# test_consumer.consumer_00()
# Publish / subscribe mode (fanout)
# test_consumer.consumer_01()
# Routing mode (direct)
# test_consumer.consumer_02()
# Topic mode to realize distribution
if sys.argv[1] == '0':
test_consumer.consumer_03_0()
elif sys.argv[1] == '1':
test_consumer.consumer_03_1()
elif sys.argv[1] == '2':
test_consumer.consumer_03_2()