rabbitmq

1. concept

   Message queue is an asynchronous communication mode between services. It is an important component in distributed system. It is used in many production environments where concurrency needs to be controlled. Message queuing provides communication and coordination for these distributed applications. Currently, the message queues used more include RabbitMQ, RocketMQ, ActivateMQ, Kafka, etc.

   • Broker: simply put, it is the message queuing server entity
   • Exchange: message switch, which specifies the rules by which messages are routed to which queue
   • Queue: message queue carrier. Each message will be put into one or more queues
   • Binding: binding, which is used to bind exchange and queue according to routing rules
   • Routing Key: the routing keyword by which exchange delivers messages
   • vhost: virtual host. Multiple vhosts can be set up in a broker to separate the permissions of different users
   • Producer: a message producer is a program that delivers messages
   • Consumer: a message consumer is a program that receives messages
   • Channel: message channel. Multiple channels can be established in each connection of the client. Each channel represents a session task

1. Application scenario

   • Application decoupling: multiple users process the same message through the message queue to avoid the failure of the whole process caused by the failure of the calling interface.
   • Asynchronous processing: multiple applications process the same message in the message queue. Messages are processed concurrently between applications, which reduces the processing time compared with serial processing
   • Current limiting and peak shaving: Rush buying activities such as double 11 and 618.
   • Message driven system: the business volume continues to expand, adopting the design idea of micro service and distributed deployment.

   Cost:

   • Application complexity: message queues need to be managed
   • Temporary inconsistency

   Use message queuing to meet the following conditions:

   • Producers do not need immediate feedback from consumers
   • Allow transient inconsistencies
   • It plays the role of decoupling, speed-up, broadcasting, peak shaving, etc

   What is the usage scenario of message queue?

Docker installation

Rabbitmq installs rabbitmq management on the portal platform.

1. visit dockerhub , search rabbitmq and click rabbitmq to get the dockerfile link of rabbitmq management.

FROM rabbitmq:3.9

RUN set eux; \
	rabbitmq-plugins enable --offline rabbitmq_management; \
# make sure the metrics collector is re-enabled (disabled in the base image for Prometheus-style metrics by default)
	rm -f /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf; \
# grab "rabbitmqadmin" from inside the "rabbitmq_management-X.Y.Z" plugin folder
# see https://github.com/docker-library/rabbitmq/issues/207
	cp /plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin /usr/local/bin/rabbitmqadmin; \
	[ -s /usr/local/bin/rabbitmqadmin ]; \
	chmod +x /usr/local/bin/rabbitmqadmin; \
	apt-get update; \
	apt-get install -y --no-install-recommends python3; \
	rm -rf /var/lib/apt/lists/*; \
	rabbitmqadmin --version

EXPOSE 15671 15672

1. Create a mirror on Protainer.

1. Running the image mainly maps 15672 (management port) and 5672 (amqp port) of the container.

2. Finally, access http: / / [server ip]:15672 to the rabbitmq management interface, and enter the default account password guest/guest to access.

3. About rabbitmq management interface
   

   

   

   

Click any Exchange:

Click any queue:

Use an example of a post office to illustrate their respective functions. First, the post office represents a queue, and the mailbox is a channel. The function of channel is to establish session tasks. It's "expensive" to set up a post office in each place. Similarly, it's very "expensive" and time-consuming to set up TCP/IP links every time. Users don't need to go to the post office every time. They just need to put the letter in the mailbox. After receiving the user's letter, the post office delivers it to the recipient according to the address (exchange) on the envelope.

python implementation

1. Simple consumer producer model

import pika
import json

credentials = pika.PlainCredentials(user, user)  # mq user name and password
# The virtual queue requires the parameter virtual to be specified_ Host. If it is the default, you can leave it blank.
connection = pika.BlockingConnection(pika.ConnectionParameters(host = host,port = 5672,virtual_host = '/',credentials = credentials))
channel=connection.channel()
# Declare a message queue in which messages will be delivered. If it does not exist, it will be created
result = channel.queue_declare(queue = 'python-test')

for i in range(10):
    message=json.dumps({'OrderId':"1000%s"%i})
# Insert numeric routing into the queue_ Key is the queue name
    channel.basic_publish(exchange = '',routing_key = 'python-test',body = message)
    print('send:'+message)
connection.close()


credentials = pika.PlainCredentials('guest', 'guest')
connection = pika.BlockingConnection(pika.ConnectionParameters(host = '192.168.3.130',port = 5672,virtual_host = '/',credentials = credentials))
channel = connection.channel()
# Declare the message queue in which messages are delivered. If it does not exist, create the queue
channel.queue_declare(queue = 'python-test', durable = False)
# Define a callback function to process messages in the message queue. Here, print them out
def callback(ch, method, properties, body):
    ch.basic_ack(delivery_tag = method.delivery_tag)
    print('receive:'+body.decode())

# Tell rabbitmq to use callback to receive messages
channel.basic_consume('python-test',callback)
# Start receiving information and enter the blocking state. Only when there is information in the queue can callback be called for processing
channel.start_consuming()

Comment out the consumption code. Let's take a look at rabbitmq management

import pika
import json

credentials = pika.PlainCredentials(user, user)  # mq user name and password
# The virtual queue requires the parameter virtual to be specified_ Host. If it is the default, you can leave it blank.
connection = pika.BlockingConnection(pika.ConnectionParameters(host = host,port = 5672,virtual_host = '/',credentials = credentials))
channel=connection.channel()
# Declare a message queue in which messages will be delivered. If it does not exist, it will be created
result = channel.queue_declare(queue = 'python-test')

for i in range(10):
    message=json.dumps({'OrderId':"1000%s"%i})
# Insert numeric routing into the queue_ Key is the queue name
    channel.basic_publish(exchange = '',routing_key = 'python-test',body = message)
    print('send:'+message)
connection.close()


# credentials = pika.PlainCredentials('guest', 'guest')
# connection = pika.BlockingConnection(pika.ConnectionParameters(host = '192.168.3.130',port = 5672,virtual_host = '/',credentials = credentials))
# channel = connection.channel()
# # Declare the message queue in which messages are delivered. If it does not exist, create the queue
# channel.queue_declare(queue = 'python-test', durable = False)
# # Define a callback function to process messages in the message queue. Here, print them out
# def callback(ch, method, properties, body):
#     ch.basic_ack(delivery_tag = method.delivery_tag)
#     print('receive:'+body.decode())

# # Tell rabbitmq to use callback to receive messages
# channel.basic_consume('python-test',callback)
# # Start receiving information and enter the blocking state. Only when there is information in the queue can callback be called for processing
# channel.start_consuming()

ACK modewhen Automatic Ack mode is selected, the consumption message is automatically confirmed

1. Working mode

In one to many mode, a producer, multiple consumers and a queue, each consumer obtains a unique message from the queue. There are two message distribution mechanisms, polling distribution and fair distribution: polling distribution is characterized by sending messages to each consumer in turn. In practice, it is inevitable for multiple consumers to process some quickly and some slowly. If they have to wait until one consumer finishes processing before sending messages to the next consumer, the efficiency will be greatly reduced. The characteristic of fair distribution is that as long as consumers handle it, they will send the message to the currently idle consumers, so as to improve the consumption efficiency.

# producer

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='task_queue', durable=True)

message = ' '.join(sys.argv[1:]) or "Hello World!"
channel.basic_publish(
    exchange='',
    routing_key='task_queue',
    body=message,
    properties=pika.BasicProperties(
        delivery_mode=2,  # make message persistent
    ))
print(" [x] Sent %r" % message)
connection.close()

# consumer

import pika
import time

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='task_queue', durable=True)
print(' [*] Waiting for messages. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    time.sleep(body.count(b'.'))
    print(" [x] Done")
    ch.basic_ack(delivery_tag=method.delivery_tag)


# Fair distribution
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=callback)

channel.start_consuming()

1. Publish/Subscribe mode

publish.py

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='logs', exchange_type='fanout')

message = ' '.join(sys.argv[1:]) or "info: Hello World!"
channel.basic_publish(exchange='logs', routing_key='', body=message)
print(" [x] Sent %r" % message)
connection.close()

Subscribe.py

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='logs', exchange_type='fanout')

result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue

channel.queue_bind(exchange='logs', queue=queue_name)

print(' [*] Waiting for logs. To exit press CTRL+C')

def callback(ch, method, properties, body):
    print(" [x] %r" % body)

channel.basic_consume(
    queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

1. Routing mode

producer.py

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='direct_logs', exchange_type='direct')

severity = sys.argv[1] if len(sys.argv) > 1 else 'info'
message = ' '.join(sys.argv[2:]) or 'Hello World!'
channel.basic_publish(
    exchange='direct_logs', routing_key=severity, body=message)
print(" [x] Sent %r:%r" % (severity, message))
connection.close()

consumer.py

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='direct_logs', exchange_type='direct')

result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue

severities = sys.argv[1:]
if not severities:
    sys.stderr.write("Usage: %s [info] [warning] [error]\n" % sys.argv[0])
    sys.exit(1)

for severity in severities:
    channel.queue_bind(
        exchange='direct_logs', queue=queue_name, routing_key=severity)

print(' [*] Waiting for logs. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))


channel.basic_consume(
    queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

1. Topic mode

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='topic_logs', exchange_type='topic')

routing_key = sys.argv[1] if len(sys.argv) > 2 else 'anonymous.info'
message = ' '.join(sys.argv[2:]) or 'Hello World!'
channel.basic_publish(
    exchange='topic_logs', routing_key=routing_key, body=message)
print(" [x] Sent %r:%r" % (routing_key, message))
connection.close()

import pika
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='topic_logs', exchange_type='topic')

result = channel.queue_declare('', exclusive=True)
queue_name = result.method.queue

binding_keys = sys.argv[1:]
if not binding_keys:
    sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
    sys.exit(1)

for binding_key in binding_keys:
    channel.queue_bind(
        exchange='topic_logs', queue=queue_name, routing_key=binding_key)

print(' [*] Waiting for logs. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))


channel.basic_consume(
    queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

1. Routing mode

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))

channel = connection.channel()

channel.queue_declare(queue='rpc_queue')

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n - 1) + fib(n - 2)

def on_request(ch, method, props, body):
    n = int(body)

    print(" [.] fib(%s)" % n)
    response = fib(n)

    ch.basic_publish(exchange='',
                     routing_key=props.reply_to,
                     properties=pika.BasicProperties(correlation_id = \
                                                         props.correlation_id),
                     body=str(response))
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='rpc_queue', on_message_callback=on_request)

print(" [x] Awaiting RPC requests")
channel.start_consuming()

import pika
import uuid

class FibonacciRpcClient(object):

    def __init__(self):
        self.connection = pika.BlockingConnection(
            pika.ConnectionParameters(host='localhost'))

        self.channel = self.connection.channel()

        result = self.channel.queue_declare(queue='', exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(
            queue=self.callback_queue,
            on_message_callback=self.on_response,
            auto_ack=True)

    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    def call(self, n):
        self.response = None
        self.corr_id = str(uuid.uuid4())
        self.channel.basic_publish(
            exchange='',
            routing_key='rpc_queue',
            properties=pika.BasicProperties(
                reply_to=self.callback_queue,
                correlation_id=self.corr_id,
            ),
            body=str(n))
        while self.response is None:
            self.connection.process_data_events()
        return int(self.response)


fibonacci_rpc = FibonacciRpcClient()

print(" [x] Requesting fib(30)")
response = fibonacci_rpc.call(30)
print(" [.] Got %r" % response)

reference resources

RabbbitMq official website

Message queue rabbitmq

Six modes of rabbitmq implemented in python

Pika API document