Programmer Think

Today, I'd like to talk a little bit about JSON data processing. In recent work, because we want to update the database data to elastic search in real time, we encounter some problems of JSON data processing in practice. real-time data The real-time data acquisition is realized by the open-source canal component of Alibaba, and transmitted to t ...

A Flink program for Kafka data consumption, the Flinon Yarn model, was released in the test and production environments before. It was normal and had no problems. However, after restarting the test environment, it was redistributed again. The error was reported as follows: 2019-07-01 15:19:25,984 INFO ...

Background Ali's RicketMQ has been castrated in previous versions and re-added in the new version to solve the problem that small companies are unable to make reliable messaging middleware products. At the same time, RocketMQ also refers to the Kafka implementation, and its performance is very good. Version 2 <dependency> <groupId& ...

Some related tests were carried out for spark: wordcount test for spark, feasible test for spark streaming, and test for Kafka message production. 6.1 spark word count test The spark test case is used to test whether spark can be operated or not. import org.apache.spark.{SparkConf, SparkContext} object ...

In the previous article EFK follow-up of log system: monitor alarm monitoring In this paper, we have basically completed the construction and testing of efk monitoring and alarm system. Then, when we switch the log source to online log, there are big problems. The CPU usage rate of fluentd is still high, and the alarm message in kafka is growin ...

Opening a picture, the reader is happier, not to mention the structure chart.This schematic diagram was introduced in the previous article "kafka Producer's Storage Pool Mechanism". In the previous article, we introduced the message collection process (we became the "Storage Pool" mechanism) in this diagram. Here we introduc ...

Kafka's background knowledge has been talked about a lot, let's start to practice now, assuming you don't have Kafka and Zoo Keeper environments. Step 1: Download the code Download version 1.1.0 and unzip it. > tar -xzf kafka_2.12-2.3.0.tgz > cd kafka_2.12-2.3.0 Step 2: Start the service Running kafka requir ...

Common terms of Kafka Broker: Kafka's server is an instance of Kafka. The Kafka cluster consists of one or more Brokers, which are responsible for receiving and processing client requests. Topic: Topic, a logical container for messages in Kafka. Each message published to Kafka has a corresponding logical container, which is often used to differ ...

Version 1 CentOS 7.5 zookeeper-3.4.12 kafka _2.12-1.1.0 II. zookeeper Installation 1. Download and decompress zookeeper compression package   tar -zvxf zookeeper-3.4.12.tar.gz   2. Creating Data and Log Folders mkdir /usr/local/zookeeper-3.4.12/data mkdir /usr/local/zookeeper-3.4.12/logs 3. Copy configuration files Go to th ...

The last section describes how to submit consumer displacement. It is with the persistence of consumer displacement that consumers can continue to consume according to the previously stored consumption displacement when they are closing, rushing or e ...