Flume collection 2-Flume introduction

I Flume installation deployment Installation address: Flume official website Address: http://flume.apache.org/ Document viewing address : http://flume.apache.org/FlumeUserGuide.html Download address : http://archive.apache.org/dist/flume/ Installation deployment: CDH 6.3 is used locally Version 1, Flume has been installed. The instal ...

Posted by safra on Sat, 25 Dec 2021 11:49:21 +0100

Deploy HDP and HDF cluster Xunlei YYDS with Ambari after January 31, 2021

background On October 4, 2018, two big data pioneers cloudera and Hortonworks announced the equal merger. Cloudera acquired Hortonworks in the form of shares, and cloudera shareholders finally obtained 60% of the shares of the merged company. In November 2020, Cloudera announced that from January 31, 2021, all Cloudera software needs a val ...

Posted by zudo on Fri, 24 Dec 2021 08:30:42 +0100

Analysis of the flow of flink task execution

background From the perspective of source code, this paper mainly analyzes how the bottom layer of flink allocates tasks to each task executor for execution, as well as the thread model of task execution. It will involve how jobmaster distributes tasks to taskExecutor, the detailed process of task execution by taskExecutor, and the mailBox thr ...

Posted by Cliftron on Thu, 23 Dec 2021 20:59:06 +0100

Timer used in Flink program

Timer used in Flink program Sometimes, we need to use timers in computing tasks to help us deal with business, such as automatic settlement of orders? Automatic praise? Regular collection? Wait However, it should be noted that we cannot flexibly configure CRON expressions for computing tasks, but only specify the trigger time. 1, What kind o ...

Posted by LiamS94 on Thu, 23 Dec 2021 17:49:53 +0100

Introduction to spark - Spark operating environment

Reference link https://www.bilibili.com/video/BV11A411L7CK?p=11 Spark operating environment As a data processing framework and computing engine, Spark is designed to run in all common cluster environments. The mainstream environment in domestic work is yard, but the container environment is becoming more and more popular Local mode The s ...

Posted by FireyIce01 on Thu, 23 Dec 2021 14:01:46 +0100

National big data skills competition for college students (Hadoop cluster construction)

preface This blog will build Hadoop based on the data of the National College Students' big data skills competition in previous years and this year's training. There are corresponding screenshots of each step. The following blog is only used as a record of the process of building Hadoop ~ if there are deficiencies, you are welcome to point ...

Posted by johnsonzhang on Thu, 23 Dec 2021 09:25:26 +0100

Deep ploughing ElasticSearch - based on most_fields strategy to realize multi field query

1. Difference between best field and most fields 1. best_fields: When searching for specific concepts of words, such as "brown fox", phrases are more meaningful than independent words. The more words the document contains in the same field, the better. The score also comes from the most matching field. 2. most_fields: In order to ...

Posted by rowanparker on Wed, 22 Dec 2021 20:17:16 +0100

Zookeeper's Java API operations: create and delete nodes

catalogue 1, Environment construction 2, API operation Zookeeper 1. Connect Zookeeper preface Connect Zookeeper 2. Create node preface Create node 3. Delete node preface Delete node 1, Environment construction Create a normal Maven projectImport log4j The properties log file to the root directory of the project or under the resour ...

Posted by kalebaustin on Tue, 21 Dec 2021 15:33:45 +0100

jdk8, [one step teaching, one step in place]

public class CollectorsDemo { public static void main(String[] args) { List<Student> studentList = new ArrayList<>(); studentList.add(new Student("Zhang San", 23)); studentList.add(new Student("Li Si", 20)); studentList.add(new Student("Wang Wu", 19)); studentList.add(new Student("X ...

Posted by Valkrin on Tue, 21 Dec 2021 13:11:02 +0100

Actual combat: flink1 12 asynchronous IO access to external data - Mysql

WeChat official account: big data development and operation framework Follow to learn more about big data. For official account, please leave a message. If you think "big data development operation and maintenance architecture" is helpful to you, please forward it to the circle of friends Copy from WeChat official account, some of ...

Posted by FURQAN on Tue, 21 Dec 2021 04:23:48 +0100