Programmer Think

Author: Wu Yuntao, senior engineer of Tencent CSIG Introduction Recently, I sorted out how to use Flink to realize real-time statistics of UV and PV indicators, and communicated with colleagues in the micro vision Department of the company. Then we simplified the scenario and found that it would be more convenient to use Flink SQL to realize t ...

Make complaints about big data left and right hands plus technology Tucao group to get more information preface As an object-oriented functional programming language, Scala combines object-oriented programming with functional programming to make the code more concise, efficient and easy to understand. That's why Scala is popular. As a ...

Series contents: Look at the flink source code and learn flink Look at the flink source code and learn the flink --- flink state preface Tip: Here you can add the general contents to be recorded in this article: For example, with the continuous development of artificial intelligence, machine learning technology is becoming more and m ...

get ready final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.registerJobListener(new JobListener() { @Override public void onJobSubmitted(@Nullable JobClient jobClient, @Nullable Throwable throwable) { Logger.getLogger("test").info("onJobSubmitted"); } @Override public ...

1, Introduction The latest version 0.9 of hudi came out in September after many calls. hudi can store massive data on the basis of hadoop. It can not only batch process, but also stream process on the data lake, that is, the combination of offline and real-time. It also provides two native semantics: 1) Update/Delete records: that is, rec ...

1.Four cornerstones of Flink Flink can be so popular without its four most important cornerstones: Checkpoint, State, Time, Window. ◼ Checkpoint This is one of Flink's most important features. Flink implements a distributed and consistent snapshot based on Handy-Lamport algorithm, which provides consistent semantics. The Chandy-Lamport a ...

When the requirements of an application are relatively simple, there may not be many operators involved in data conversion, but when the requirements of the application become more and more complex, the number of operators in a Job may reach dozens or even hundreds. With so many operators, the whole application will become very complex, So it w ...