Flink (56): FlinkSQL integration Hive of Flink advanced features

catalogue 0. Links to related articles 1. Introduction to flinksql integration Hive 2. Basic ways of integrating Hive 2.1. Persistent metadata 2.2. Use Flink to read and write Hive's table 3. Preparation 4. SQL CLI 5. Code demonstration 0. Links to related articles Flink article summary 1. Introduction to flinksql integration Hive ...

Posted by sd9sd on Wed, 02 Feb 2022 20:39:17 +0100

spark the way of God - detailed explanation of RDD creation

3.2 RDD programming In Spark, RDD is represented as an object, and RDD is converted through method calls on the object. After defining RDD through a series of transformations, you can call actions to trigger the calculation of RDD. Actions can be to return results to the application (count, collect, etc.) or to save data to the storage system ...

Posted by Matty999555 on Tue, 01 Feb 2022 16:47:04 +0100

Installation and deployment of Flink

Installation and deployment of Flink First, open the terminal and execute the command under / home/xxx / to obtain the installation package: wget https://labfile.oss.aliyuncs.com/courses/3423/flink-1.10.0-bin-scala_2.11.tar tar -xvf flink-1.10.0-bin-scala_2.11.tar Unzip it to this directory. If you have studied Spark, you should feel that t ...

Posted by GESmithPhoto on Mon, 31 Jan 2022 13:56:59 +0100

Flink (51): broadcast state of Flink advanced features

catalogue 0. Links to related articles 1. Introduction to broadcaststate 2. Requirements - realize dynamic update of configuration 3. Coding steps 4. Code implementation 0. Links to related articles Flink article summary 1. Introduction to broadcaststate During the development process, if low throughput events such as distribution / b ...

Posted by Fsoft on Sat, 29 Jan 2022 06:36:12 +0100

Lesson 05: Flink SQL & Table programming and cases

Flink series Lesson 01: Flink's application scenario and architecture model Lesson 02: introduction to Flink WordCount and SQL implementation Lesson 03: Flink's programming model compared with other frameworks Lesson 04: Flink's commonly used DataSet and datastream APIs Lesson 05: Flink SQL & Table programming and cases Lesson 06: Flink cl ...

Posted by superstar on Thu, 27 Jan 2022 23:51:56 +0100

About getting started with the flow processing framework Flink

1. What is flick flink is a flow processing framework. Usually, the usage scenario is to consume kafka data and send it to other systems after grouping and aggregation. Grouping and aggregation are the core of flink. This paper only describes a single usage scenario. Stream data is equivalent to continuous data. The log data in kafka in produc ...

Posted by Roo on Wed, 26 Jan 2022 23:42:23 +0100

Flink1.12 - JobManager&TaskManager memory management

Flink1.12 - memory management 1. Foreword In order to enable users to better adjust memory allocation and achieve reasonable resource allocation, flynk1 10 introduces the memory management of TaskManager, which is later described in flink1 Version 11 introduces the memory management of JobManager. Users can allocate resources reasonably th ...

Posted by altemur on Mon, 24 Jan 2022 04:23:38 +0100

Flink (48): TableAPI for Flink and common operators in FlinkSQL

Catalog 0. Links to related articles 1. select operator 2. where operator 3. distinct operator 4. group by operator 5. union and union all operators 6. join operator 7. group window operator 7.1. tumble window operator 7.2. hop window operator 7.3. session window operator 0. Links to related articles Flink Article Summary 1. sele ...

Posted by jehardesty on Sun, 23 Jan 2022 00:20:52 +0100

Flink common operators

DataStream common operators In the Flink application, whether your application is a batch program or a stream program, it is the model shown in the figure above. There are data sources and data sink s. The applications we write mostly do a series of operations on the data from the data source, which are summarized as follows. Source: da ...

Posted by rklapwijk on Sat, 22 Jan 2022 05:01:01 +0100

[Flink] [Chapter 8] ProcessFunctionAPI

1. Introduction to processfunction 1.1 description on API A function that processes elements of a stream. For every element in the input stream processElement(Object, ProcessFunction.Context, Collector) is invoked. This can produce zero or more elements as output. Implementations can also query the time and set timers through the p ...

Posted by sleepydad on Fri, 21 Jan 2022 09:34:20 +0100