Index statistics: real time UVPV statistics based on flow computing Oceanus(Flink)

Author: Wu Yuntao, senior engineer of Tencent CSIG Introduction Recently, I sorted out how to use Flink to realize real-time statistics of UV and PV indicators, and communicated with colleagues in the micro vision Department of the company. Then we simplified the scenario and found that it would be more convenient to use Flink SQL to realize t ...

Posted by kula on Fri, 22 Oct 2021 07:29:47 +0200

Scala essence is here, take it, interview is not afraid.

Make complaints about big data left and right hands plus technology Tucao group to get more information preface As an object-oriented functional programming language, Scala combines object-oriented programming with functional programming to make the code more concise, efficient and easy to understand. That's why Scala is popular. As a ...

Posted by Jak on Mon, 11 Oct 2021 03:38:55 +0200

Look at the flink source code and learn the flink --- flink state

Series contents: Look at the flink source code and learn flink Look at the flink source code and learn the flink --- flink state preface Tip: Here you can add the general contents to be recorded in this article: For example, with the continuous development of artificial intelligence, machine learning technology is becoming more and m ...

Posted by annihilate on Fri, 08 Oct 2021 11:14:35 +0200

Flink DataStream

get ready final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.registerJobListener(new JobListener() { @Override public void onJobSubmitted(@Nullable JobClient jobClient, @Nullable Throwable throwable) { Logger.getLogger("test").info("onJobSubmitted"); } @Override public ...

Posted by Corvin on Fri, 08 Oct 2021 11:13:12 +0200

The practice of data Lake based on flink+hudi+hive

1, Introduction The latest version 0.9 of hudi came out in September after many calls. hudi can store massive data on the basis of hadoop. It can not only batch process, but also stream process on the data lake, that is, the combination of offline and real-time. It also provides two native semantics: 1) Update/Delete records: that is, rec ...

Posted by carnold on Mon, 27 Sep 2021 14:43:07 +0200

Large Data Flink Window Operation

1.Four cornerstones of Flink Flink can be so popular without its four most important cornerstones: Checkpoint, State, Time, Window. ◼ Checkpoint This is one of Flink's most important features. Flink implements a distributed and consistent snapshot based on Handy-Lamport algorithm, which provides consistent semantics. The Chandy-Lamport a ...

Posted by rvpals on Fri, 10 Sep 2021 01:37:26 +0200

How to view Flink job execution plan

When the requirements of an application are relatively simple, there may not be many operators involved in data conversion, but when the requirements of the application become more and more complex, the number of operators in a Job may reach dozens or even hundreds. With so many operators, the whole application will become very complex, So it w ...

Posted by navtheace on Sun, 05 Sep 2021 02:17:18 +0200