Usually in the Pulsar communication group, we find that you will encounter similar problems repeatedly in the process of contacting and using Pulsar. In order to solve these "high frequency questions" more efficiently, and to thank the friends who have asked good questions, we have set up a FAQ knowledge base to collect and answer your questions.
We will regularly collect high-frequency questions from the screening community, and select high-quality questions and answers from them by community experts. We will integrate and optimize the questions and share them with our community partners as a priority reference when we encounter problems. We hope to help you solve the problems in using Pulsar.
Let's take a look at this collection of questions:
Message Retention
Question 1: In Pulsar's message retention mechanism, Pulsar's message will enter the retention policy after it is Ack by a consumer. Does this message enter a retention policy when multiple consumers subscribe to a Topic, part of a message, Ack?
Answer: Only if the message is ACK in all subscriptions will it be further processed by the retention policy. For the same subscription, it does not matter which consumer performs the ACK. As long as it is ACK once, the message in this subscription is already ACK. Specific reference: https://pulsar.apache.org/doc...
retry count
Question 2: Does Pulsar support parameter configuration retries?
Answer: Not supported. Pulsar evades retry policies by time, similar to the retry count mechanism. The retry policy inside the client is a backoff mechanism that can be controlled by configuring the backoff parameter of ClientBuilder.
/**
* Set the duration of time for a backoff interval.
*
* @param duration the duration of the interval
* @param unit the time unit in which the duration is defined
* @return the client builder instance
*/
ClientBuilder startingBackoffInterval(long duration, TimeUnit unit);
/**
* Set the maximum duration of time for a backoff interval.
*
* @param duration the duration of the interval
* @param unit the time unit in which the duration is defined
* @return the client builder instance
*/From a producer's perspective, you can configure the total send timeout in ProducerBuilder.
/**
* Set the send timeout <i>(default: 30 seconds)</i>.
*
* <p>If a message is not acknowledged by the server before the sendTimeout expires, an error will be reported.
*
* <p>Setting the timeout to zero, for example {@code setTimeout(0, TimeUnit.SECONDS)} will set the timeout
* to infinity, which can be useful when using Pulsar's message deduplication feature, since the client
* library will retry forever to publish a message. No errors will be propagated back to the application.
*
* @param sendTimeout
* the send timeout
* @param unit
* the time unit of the {@code sendTimeout}
* @return the producer builder instance
*/
ProducerBuilder<T> sendTimeout(int sendTimeout, TimeUnit unit)Pulsar Perf
Question 3: Pulsar Perf is used to produce data, but there is no backlog.
Answer: Create Subscription before Pulsar Perf production data.
Pressure tool
Question 4: How to test Pulsar? What tools are used?
Answer: Pulsar Pressure Tool currently has Pulsar Perf and OpenMessaging Benchmark, refer to the following links for details:
Message Delay
Question 5: Does the Pulsar server support delayed delivery by default?
Answer: delay message only works in Shared subscription mode. If it doesn't work, you need to check first if the subscription mode is Shared and Pulsar itself defaults to Exclusive subscription mode. delay message and Key_should also be avoided from taking effect in KeyShared mode Shared semantics are violated.
Broker Downtime
Question 6: If the broker is down, how can the message be notified to the next broker?
Answer: Broker itself is stateless.
- Broker does not actively notify other brokers when it is down. All bundles distributed on this broker (bundles are topic collections, the smallest unit of Pulsar topic load balancing) perform unload. The unload process:
- Turn off all topic producers, consumers and replicators in unload;
- Close the Managedledger. When all bundle s are unload ed, the broker can exit normally.
- If the Broker exits abnormally, all bundles on the Broker will also be forcibly removed from the corresponding attribution, that is, the bundle does not belong to the Broker.
When a producer or consumer client needs to continue sending/receiving messages to a top, the lookup request is executed first, the target broker node (currently the lowest loaded node) is found according to the loadbalance policy, and the corresponding bundle onLoad is sent to the target broker. When onLoad is finished, the broker can continue to provide read and write services to the top.
In addition, some temporary node information of Broker on ZooKeeper will be deleted actively; Or after a time-out disconnects and other brokers listen, they will do so.
Compact Limit
Question 7: Disk occupancy is so high that disk cleaning is much slower than writing. Adjusting the BookKeeper parameters gcWaitTime, majorCompactionInterval, minorCompactionInterval is not effective. Is there another solution?
Answer: Compaction has a current limiting strategy. From your description, Compaction should be slower. You can adjust isThrottleBytes=true and increase the current limiting threshold of compactionRateBytes.
# Throttle compaction by bytes or by entries. isThrottleByBytes=false # Set the rate at which compaction will readd entries. The unit is adds per second. compactionRateByEntries=1000 # Set the rate at which compaction will readd entries. The unit is bytes added per second. compactionRateByBytes=1000000
Pulsar Permissions
Question 8: Pulsar privilege related data.
Answer:
This is the fourth FAQ summary of the community. Thank you for your participation in the community's daily questions and answers. Let's look forward to the next FAQ!