Programmer Think

preface Usually, in our project development, we often encounter the situation that we need to automatically execute some business code as soon as the service is started. For example, cache the configuration information or data dictionary in the database to redis, or open some configured scheduled tasks when the service is started. There are act ...

Chapter 3 DStream creation 3.1 RDD queue 3.1.1 usage and description During the test, you can create a DStream by using ssc.queueStream(queueOfRDDs). Each RDD pushed to the queue will be processed as a DStream. 3.1.2 case practice Requirement: create several RDD S in a loop and put them into the queue. Create Dstream through SparkSt ...

1. BackgroundThe Proxy service is responsible for consuming and parsing CKafka messages, and distributing messages to different CKafka topic s. Recently, it was found that the Proxy service consumption CKafka has message accumulation, and the CVM CPU and memory resources where the service is located only occupy about 50%.As shown in the figure, ...

1 Introduction When it comes to message oriented middleware, I think everyone should be familiar with it and have more or less contact with it. In fact, the popular understanding is that message middleware is a developed system and deployed independently, and then our business system sends and receives messages through it, so as to achieve t ...

Set up hadoop,zookeeper,kafka in the environment of data warehouse collection Previous Aliyun ECS environment has been installed, now start to formally set up the project environment hadoop installation configuration 1. Cluster Planning Server hadoop102Server hadoop103Server hadoop104HDFSNameNodeDataNodeDataNodeDataNodeSecondaryNameNod ...

1.jdk installation 1) windows install jdk 1. Download Open the following link: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html , enter the JDK1.8 download official website, or directly Baidu JDK1.8, or enter the download official website. After entering the official website, download the JDK of the c ...

In the previous article, we learned about specific terms in Kafka and related topic concepts, which you can refer to in more detail Kafka Learning Notes (2): Understanding Kafka Cluster and Topic Next we'll use the Java language to call Kafka's API. Today we'll first look at what we can do with Producer. Add Dependency First let's add maven d ...

Preface Functional debugging is often done in collaboration with some software during development, for example, when working with Flink CDC, you need to import data from mysql binlog into kafka and then into hudi data lake. So the problem is here. To do this, I need to start with a mysql, a kafka, a yarn cluster, a hdfs cluster so that the wh ...

Software is becoming more and more complex, which has become a trend. The traditional process oriented method can not meet the needs of software development. In order to better maintain the software, people begin to consider corresponding the real-world entities with the modules in the software. The modules in the software not only have the ...

Summary of practical applications of Kafka, RabbitMQ and RockedMQ 1.Kafka Combined with the use cases on the official website, this paper records the examples and practical application of the three mainstream mq.This article does not cover the installation and configuration of relevant environments, but involves more comprehensive codes (inclu ...