Learning notes of Java Engineer Interview assault Season 1 - message queue 09, 10, 11
Only to make personal learning records, the content comes from the teaching of the stone shirt code farmer's official account.
Learning purpose:
(1) Help quickly sort out the knowledge points of high-frequency Java advanced interview of Internet companies;
(2) Help quickly consolidate the knowledge system of Java advanced technology stack;
(3) Go out for an interview after learning, and be able to hold the serial gun questions of some Internet companies on a technical point;
catalog:
09 how can I ensure that the data obtained from the message queue is executed in order?
How to ensure the order of messages?
10 over! Production accident! Millions of messages have been backlog in the message queue for hours!
How to ensure that messages are not consumed repeatedly (how to ensure idempotency when messages are consumed)?
If you were asked to develop a Message Queuing Middleware, how would you design the architecture?
If you are asked to write a message queue, how to design the architecture? Tell me what you think
--------------------------------------------------------Text----------------------------------------------
09 how can I ensure that the data obtained from the message queue is executed in order?
1. Interview questions
How to ensure the order of messages?
2. Interviewer psychological analysis
In fact, this is also used MQ First, let's see if you understand the order? Second, see if you can ensure that the messages are in order? Common problems in this production system.
3. Analysis of interview questions
Let me give you an example. We've done one before mysql binlog+The pressure of synchronous system is still very large, and the daily synchronous data should reach hundreds of millions. mysql -> mysql，A common point is that big data team，You need to synchronize one mysql Library to do all kinds of complex operations on the data of the company's business system. you are here mysql One piece of data is added, deleted and modified in the, and three pieces of data are added, deleted and modified binlog，Then these three binlog Send to MQ Inside, it is implemented in turn from consumption to consumption. At least we have to ensure that others come in order, right? Otherwise, it is: add, modify, delete; You changed the order to delete, modify and add. Aren't they all wrong. Originally, this data was synchronized, but it should be deleted at last; As a result, you make a mistake in this order. Finally, the data is retained, and the data synchronization is wrong.
Let's take a look at the two scenes that will be out of order
(1) rabbitmq: one queue and multiple consumer s. This is not obviously messy
(2) kafka: a topic, a partition, a consumer, and internal multithreading. It's obviously messy
How to ensure the order of messages? Simple simple
(1) rabbitmq: split multiple queues, one consumer for each queue, just more queues, which is really troublesome; Or a queue corresponds to a consumer, which is internally queued with a memory queue, and then distributed to different underlying worker s for processing
img
(2) kafka: a topic, a partition, and a consumer. Internal single thread consumption. Write N memory queues, and then N threads consume one memory queue respectively
img
10 over! Production accident! Millions of messages have been backlog in the message queue for hours!
1, Interview questions
How to solve the problem of message queue delay and expiration? What should I do when the message queue is full? There are millions of messages that are kept in backlog for several hours. How to solve them?
2, Interviewer psychological analysis
You see, this question is essentially aimed at the scene, which means that there may be something wrong with your consumption, you don't consume, or your consumption is extremely slow. Then there is the pit father. Maybe the disks of your message queue cluster are almost full and no one consumes them. What should I do at this time? Or the whole has been overstocked for several hours. What do you do at this time? Or your backlog is too long, resulting in, for example rabbitmq What if there is no message after the message expiration time is set? So it's just this. In fact, it's very common online. Generally, it's not out. Once it's out, it's big case，It is common in, for example, the consumer needs to write after each consumption mysql，result mysql Hang up, consumer hang There it is. It doesn't move. Or the consumer brought out a fork, resulting in extremely slow consumption.
3, Analysis of interview questions
Let's sort this out one by one. Let's assume a scenario. We now have a consumer failure, and then a lot of messages are on the Internet mq There is a backlog in the. Now there is an accident. I'm in a panic
1. A large number of messages have been overstocked in mq for several hours and haven't been solved yet
Tens of millions of data in MQ There was a backlog of seven or eight hours, from more than 4 p.m. to late at night, more than 10 p.m. and more than 11 p.m. This is a real scene we have encountered. It is indeed an online fault. At this time, it is either repaired consumer Let him restore the consumption speed, and then wait for a few hours to finish the consumption. This certainly can't be said during the interview. A consumer has 1000 items per second, 3000 items per second for three consumers, 180000 items per minute, and more than 10 million items. So if you have a backlog of millions to tens of millions of data, even if consumers recover, it will take about an hour to recover.
Generally, only temporary emergency capacity expansion can be operated at this time. The specific operation steps and ideas are as follows:
(1) First fix the problem of consumer s to ensure their consumption speed is restored, and then stop all existing cnosumer s;
(2) Create a new topic with 10 times the original partition, and temporarily create 10 times or 20 times the original number of queue s;
(3) Then write a temporary consumer program for distributing data. This program is deployed to consume the backlog of data. After consumption, it does not do time-consuming processing, and directly polls and writes 10 times the number of queue s established temporarily;
(4) Then temporarily requisition 10 times of machines to deploy consumers, and each batch of consumers consumes data of a temporary queue;
(5) This approach is equivalent to temporarily expanding queue resources and consumer resources by 10 times to consume data at a normal rate of 10 times;
(6) After quickly consuming the backlog data, we have to restore the original deployment architecture and reuse the original consumer machine to consume messages;
Insert picture description here
2. Here, let's assume a second pit
Suppose you use rabbitmq，rabbitmq You can set the expiration time, that is TTL，If the message is queue If the backlog exceeds a certain time, it will be destroyed rabbitmq Clean it up and the data will be gone. Then this is the second pit. This is not to say that there will be a large backlog of data mq But a lot of data will be lost directly. In this case, it does not mean to increase consumer Consumption backlog of news, because in fact there is no backlog, but lost a lot of news. We can adopt a scheme, that is, batch homing, which has been done in similar scenes on our front line. When there is a large backlog, we directly discard the data at that time, and then wait after the peak period. For example, we drink coffee together and stay up late until after 12 p.m., and users go to bed. At this time, we began to write programs, write a temporary program to find out the lost batch of data bit by bit, and then refill it mq Go inside and make up for the data lost during the day. That's the only way. Suppose 10000 orders are overstocked mq Inside, there was no processing. Among them, 1000 orders were lost. You can only manually write a program to find out the 1000 orders and manually send them to mq Go inside and make it up again.
3. Then let's assume the third pit
If the way to go is message backlog mq If you haven't disposed of it for a long time, it will lead to mq It's almost full. What should I do? Is there any other way to do this? No, who told you to execute the first scheme too slowly? You write programs temporarily, access data to consume, consume one, discard one, don't want any, and consume all messages quickly. Then take the second plan and supplement the data in the evening.
If you were asked to develop a Message Queuing Middleware, how would you design the architecture?
1. Interview questions
If you are asked to write a message queue, how to design the architecture? Tell me what you think
2. Psychological analysis of interviewers
In fact, when talking about this problem, the general interviewer should examine two aspects:
(1) Have you ever had a deeper understanding of the principle of a message queue, or grasped the architecture principle of an mq from the overall understanding
(2) Take a look at your design ability and give you a common system, message queuing system. See if you can grasp the overall architecture design from the overall situation and give some key points
To tell you the truth, when I face similar problems, most people basically get confused, because they have never thought about similar problems. Most people are usually buried in using them and never think about something behind them. I often ask similar questions. What would you do if you were asked to design a spring framework? What would you do if you were asked to design a dubbo framework? What would you do if you were asked to design a mybatis framework?
3. Analysis of interview questions
In fact, to answer such questions, to put it bluntly, at least you don't need to read the source code of the technology. At least you probably know the basic principle, core components and basic architecture of the technology, and then say the idea of designing a system with reference to some open source technologies
For example, the message queue system, let's consider it from the following perspectives
(1) First of all, the mq must support scalability, that is, it can increase throughput and capacity by rapidly expanding capacity when necessary. What's the matter? Design a distributed system. Refer to kafka's design concept, broker - > topic - > partition. Each partition puts a machine and stores part of the data. If there are not enough resources now, simply add a partition to the topic, then migrate the data and add machines, can you store more data and provide higher throughput?
(2) Secondly, you have to consider whether the mq data should be on a landing disk? You must drop the disk to ensure that the process does not hang and the data is lost. How did you drop the disk? Sequential write, so there is no addressing overhead of random disk read and write. The performance of disk sequential read and write is very high. This is kafka's idea.
Secondly, you consider the availability of your mq? For this, please refer to kafka's high availability guarantee mechanism explained in the previous availability link. Multiple copies - > leader & follower - > broker hang up and re elect leaders to provide external services. (4) Can you support data 0 loss? Yes, refer to the kafka data zero loss scheme we mentioned earlier
In fact, an mq must be very complex. When the interviewer asks you this question, it is actually an open question. He just wants to see if you have the thinking and ability of overall conception and design from the perspective of architecture. Indeed, this problem can brush off a large number of people, because most people usually don't think about these things.