Flume collection 2-Flume introduction
I Flume installation deployment
Installation address:
Flume official website Address: http://flume.apache.org/ Document viewing address : http://flume.apache.org/FlumeUserGuide.html Download address : http://archive.apache.org/dist/flume/
Installation deployment: CDH 6.3 is used locally Version 1, Flume has been installed. The instal ...
Posted by safra on Sat, 25 Dec 2021 11:49:21 +0100
Deploy HDP and HDF cluster Xunlei YYDS with Ambari after January 31, 2021
background
On October 4, 2018, two big data pioneers cloudera and Hortonworks announced the equal merger. Cloudera acquired Hortonworks in the form of shares, and cloudera shareholders finally obtained 60% of the shares of the merged company. In November 2020, Cloudera announced that from January 31, 2021, all Cloudera software needs a val ...
Posted by zudo on Fri, 24 Dec 2021 08:30:42 +0100
Analysis of the flow of flink task execution
background
From the perspective of source code, this paper mainly analyzes how the bottom layer of flink allocates tasks to each task executor for execution, as well as the thread model of task execution. It will involve how jobmaster distributes tasks to taskExecutor, the detailed process of task execution by taskExecutor, and the mailBox thr ...
Posted by Cliftron on Thu, 23 Dec 2021 20:59:06 +0100
Timer used in Flink program
Timer used in Flink program
Sometimes, we need to use timers in computing tasks to help us deal with business, such as automatic settlement of orders? Automatic praise? Regular collection? Wait
However, it should be noted that we cannot flexibly configure CRON expressions for computing tasks, but only specify the trigger time.
1, What kind o ...
Posted by LiamS94 on Thu, 23 Dec 2021 17:49:53 +0100
Introduction to spark - Spark operating environment
Reference link
https://www.bilibili.com/video/BV11A411L7CK?p=11
Spark operating environment
As a data processing framework and computing engine, Spark is designed to run in all common cluster environments. The mainstream environment in domestic work is yard, but the container environment is becoming more and more popular
Local mode
The s ...
Posted by FireyIce01 on Thu, 23 Dec 2021 14:01:46 +0100
National big data skills competition for college students (Hadoop cluster construction)
preface
This blog will build Hadoop based on the data of the National College Students' big data skills competition in previous years and this year's training. There are corresponding screenshots of each step. The following blog is only used as a record of the process of building Hadoop ~ if there are deficiencies, you are welcome to point ...
Posted by johnsonzhang on Thu, 23 Dec 2021 09:25:26 +0100
Deep ploughing ElasticSearch - based on most_fields strategy to realize multi field query
1. Difference between best field and most fields
1. best_fields:
When searching for specific concepts of words, such as "brown fox", phrases are more meaningful than independent words. The more words the document contains in the same field, the better. The score also comes from the most matching field.
2. most_fields:
In order to ...
Posted by rowanparker on Wed, 22 Dec 2021 20:17:16 +0100
Zookeeper's Java API operations: create and delete nodes
catalogue
1, Environment construction
2, API operation Zookeeper
1. Connect Zookeeper
preface
Connect Zookeeper
2. Create node
preface
Create node
3. Delete node
preface
Delete node
1, Environment construction
Create a normal Maven projectImport log4j The properties log file to the root directory of the project or under the resour ...
Posted by kalebaustin on Tue, 21 Dec 2021 15:33:45 +0100
jdk8, [one step teaching, one step in place]
public class CollectorsDemo {
public static void main(String[] args) {
List<Student> studentList = new ArrayList<>();
studentList.add(new Student("Zhang San", 23));
studentList.add(new Student("Li Si", 20));
studentList.add(new Student("Wang Wu", 19));
studentList.add(new Student("X ...
Posted by Valkrin on Tue, 21 Dec 2021 13:11:02 +0100
Actual combat: flink1 12 asynchronous IO access to external data - Mysql
WeChat official account: big data development and operation framework
Follow to learn more about big data. For official account, please leave a message.
If you think "big data development operation and maintenance architecture" is helpful to you, please forward it to the circle of friends
Copy from WeChat official account, some of ...
Posted by FURQAN on Tue, 21 Dec 2021 04:23:48 +0100