Analysis of the flow of flink task execution

background From the perspective of source code, this paper mainly analyzes how the bottom layer of flink allocates tasks to each task executor for execution, as well as the thread model of task execution. It will involve how jobmaster distributes tasks to taskExecutor, the detailed process of task execution by taskExecutor, and the mailBox thr ...

Posted by Cliftron on Thu, 23 Dec 2021 20:59:06 +0100

Timer used in Flink program

Timer used in Flink program Sometimes, we need to use timers in computing tasks to help us deal with business, such as automatic settlement of orders? Automatic praise? Regular collection? Wait However, it should be noted that we cannot flexibly configure CRON expressions for computing tasks, but only specify the trigger time. 1, What kind o ...

Posted by LiamS94 on Thu, 23 Dec 2021 17:49:53 +0100

How does Flink deploy and install?

preface This article belongs to the column "1000 problems to solve big data technology system", which is original by the author. Please indicate the source of quotation. Please help point out the deficiencies and errors in the comment area. Thank you! Please refer to table of contents and references for this column 1000 problems ...

Posted by njdirt on Thu, 23 Dec 2021 00:53:55 +0100

Real time data warehouse: real time analysis of MySQL and HBase dimension tables to ClickHouse based on flow computing Oceanus

Real time is the future. Recently, we shared real-time computing services at Tencent cloud computing Oceanus (Flink)~Project backgroundThis paper introduces how to build a real-time data warehouse by combining MySQL database, stream computing Oceanus (Flink), HBase and cloud data warehouse ClickHouse, read MySQL data through stream computing Oc ...

Posted by irishmike2004 on Wed, 22 Dec 2021 19:17:55 +0100

Reading Flink source code and talking about design: the way to effectively manage memory

This article was first published in Simple book of mooring and floating purpose:https://www.jianshu.com/u/204...editiondateremarks1.02021.12.20Article launch0. PrefaceWhen I first came into contact with Flink, it was shared by some leading players in the industry - we will use it to process massive data. In this scenario, the problem of how to ...

Posted by Tryfan on Wed, 22 Dec 2021 03:42:34 +0100

flink DataStream API status and fault tolerance - broadcast status mode

Broadcast status mode In this section, you will learn how to use broadcast status in practice. See yes Stateful Stream Processing To understand the concepts behind stateful flow processing. API provided In order to demonstrate the provided API s, we will start with an example and then show their full functionality before showing their fu ...

Posted by josborne on Tue, 21 Dec 2021 14:15:08 +0100

Actual combat: flink1 12 asynchronous IO access to external data - Mysql

WeChat official account: big data development and operation framework Follow to learn more about big data. For official account, please leave a message. If you think "big data development operation and maintenance architecture" is helpful to you, please forward it to the circle of friends Copy from WeChat official account, some of ...

Posted by FURQAN on Tue, 21 Dec 2021 04:23:48 +0100

Flink development - relationship between Tasks and operator chain

For distributed execution, Flink links the subtasks of operators together to form tasks, and the operators in each subtask are connected into a chain, that is, the Operator chain. Compared with each task executed by one thread, linking operators into tasks is a useful optimization: it reduces the overhead of inter thread switching and buffe ...

Posted by eraxian on Mon, 20 Dec 2021 19:13:17 +0100

Analysis and implementation of window TopN

Statement: this series of blogs is compiled according to SGG's videos, which is very suitable for you to learn. Some articles are collected by crawlers and other technical means in order to learn and share. If there is a copyright problem, please leave a message and delete it at any time. The latest version of big data interview questions in 2 ...

Posted by pbaker on Sat, 18 Dec 2021 23:24:22 +0100

Custom metric monitors inflow and output

Statement: this series of blogs is compiled according to SGG's videos, which is very suitable for you to learn. Some articles are collected by crawlers and other technical means in order to learn and share. If there is a copyright problem, please leave a message and delete it at any time. The latest version of big data interview questions in 2 ...

Posted by mrgym on Sat, 18 Dec 2021 19:53:39 +0100