Flink1.12 - memory management

1. Foreword

In order to enable users to better adjust memory allocation and achieve reasonable resource allocation, flynk1 10 introduces the memory management of TaskManager, which is later described in flink1 Version 11 introduces the memory management of JobManager. Users can allocate resources reasonably through configuration.

Both task manager and JobManager are separate JVM processes. They share a set of memory model abstractions (the memory model of task manager is more complex), as follows.

The model consists of heap and off heap memory. There are three ways to adjust memory. One of the three ways must be selected, otherwise the startup will fail. It is best not to specify both ways at the same time, otherwise there may be configuration conflict.

for TaskManager:	for JobManager:
taskmanager.memory.flink.size	jobmanager.memory.flink.size
taskmanager.memory.process.size	jobmanager.memory.process.size
taskmanager.memory.task.heap.size and taskmanager.memory.managed.size	jobmanager.memory.heap.size

• Directly adjust jobmanager memory. process. Size or taskmanager memory. process. Size way

  • This is the simplest way. The memory allocation of all components under this configuration will have default values or be derived. It is usually specified in the case of yarn, k8s and mesos. The container deployment applies for the size of the container through the configuration of this parameter.

• Directly adjust jobmanager memory. flink. Size or taskmanager memory. flink. Size way

  • This method is usually used in standalone, and other components are not

• The third method is to directly configure the size of heap and off heap components to achieve more fine-grained control, or it can be used in combination with one of the first two methods.

Choose one of the above three methods ~!!

2. JobManager memory allocation

The memory model of JobManager is as follows

The above model diagram of total process size can be divided into the following four memory components. If one or more of the components are specified when allocating memory, the value of JVM overhead is determined by other components. The value obtained by total process size - other components must be between min and max. if the value of components is not specified, Then calculate according to the fraction of 0.1. If the calculated value is less than min, take min; if it is greater than max, take max; if min and Max are equal, then the JVM overhead is a determined value!

Memory components	configuration option	Functions of memory components
JVM Heap	jobmanager.memory.heap.size	This size depends on the number of jobs submitted, the structure of jobs and the requirements of user code. = > > > It is mainly used to run the flink framework, execute the user code when job submission and the callback code of checkpoint
Off-heap Memory	jobmanager.memory.off-heap.size (default 128M)	JM's external memory size Covers all direct and native memory allocations. = > > > > It is used to execute external dependencies such as akka, and is also responsible for running checkpoint callback and user code when job submission
JVM metaspace	jobmanager.memory.jvm-metaspace.size (256M by default)	The meta space size of JM, with the default value of jobmanager memory. jvm-metaspace. Size = 256M, belonging to native memory
JVM Overhead	jobmanager.memory.jvm-overhead.min (192M)jobmanager.memory.jvm-overhead.max (1G)jobmanager.memory.jvm-overhead.fraction(0.1)	The native memory reserved for thread stacks, code cache and garbage collection space has the default factor of total process size, but it must be between Min & max

2.1. Allocate total process size

• jobmanager.memory.process.size

# At this point, we only show the specified JobManager memory. process. Size. No other components are specified. At this time, the memory occupied by the JVM process of the entire JobManager is 2000M
0,total process size = 2000M(This is the assigned baseline value)
1,JVM overhead Because no other component memory is specified, it is set as 0.1 of fraction Infer into => 2000M * 0.1 * 1024 * 1024 = 209715203B(200M)
2,JVM Metaspace The default value is 256 M
3,Off-Heap Memeory The default is 128 M
4,JVM Heap It was finally inferred as 2000 M - 200M - 256M - 128M = 1.38G(？？？？？？？？？？？？？)

√ why does JVM Heap only have 1.33GB instead of 1.38GB?

In fact, it depends on the GC algorithm you use, which will occupy a small part of the fixed memory as non heap. The size of the occupied part is: 1.38-1.33 = 0.05GB.

2.2. Allocate total flink size

• jobmanager.memory.flink.size

# At this point, we only show the specified JobManager memory. flink. Size, and other components such as heap size are not specified. At this time, the JVM process of the entire JobManager can occupy 2000M memory except JVM Overhead and JVM Metaspace
0,total flink size = 2000M = 1.95G(This belongs to total process size One of the components of, overhead Can only be inferred from the remaining memory)
1,JVM Metaspace The default value is 256 M(Fixed (default)
2,Off-Heap Memeory The default value is 128 M(Fixed (default)
3,JVM heap = 2000M - 128M = 1.828GB - 80MB(GC Algorithm occupancy) = 1.75GB
4,according to JVM overhead  = (JVM overhead + 256M(Metaspace) + 2000(flink size)) * 0.1  
=> total process size =  2.448GB
=> JVM overhead = 2.448GB * 0.1 =  262843055B =  250.667MB(At 192 M～1GB)，As valid
5,The final resource allocation is shown in the above log～～

2.3. Allocate heap size separately

# At this point, we only show the specified JobManager memory. heap. The value of size is equivalent to displaying the value of configured components. At this time, the JVM Heap of the entire JobManager is specified as the maximum memory of 1000M
0,Jvm heap Designated as 1000 M，But we have to start from GC Deduct 41 from the algorithm MB(GC Algorithm occupancy) => JVM heap(actual) = 959MB
1,JVM Metaspace The default value is 256 M
2,Off-Heap Memeory The default value is 128 M
3,flink total size = 1128MB = 1.102GB
4,(1128MB + 256M + JVM overhead)* 0.1 = JVM overhead => JVM = 153.778 < 192MB(default min) => JVM overhead = 192MB
5,total process size = 1128MB + 256M + JVM overhead = 1576MB = 1.5390625GB = 1.539GB

2.4. Allocate process size and heap size

# Because heap. Is specified Size is the size of the memory component, so JVM overhead is to take the remaining total process size memory space
0,total process size = 2000MB && JVM heap = 1000MB,Actually, it's only 959 MB，Because minus 41 MB of GC Algorithm space
1,JVM Metaspace The default value is 256 M
2,Off-Heap Memeory The default value is 128 M
3,total flink size = 1000MB + 128MB = 1128MB
4,JVM overhead = 2000MB - 1128MB - 256MB = 616MB 

2.5. Allocate flick size and heap size

# Because head. Is specified Size component, the overhead will be allocated according to the memory space of the remaining total process size
0,total flink size = 2000MB && JVM heap = 1000MB，Actual 959 MB，Minus GC Space occupied by algorithm
1,JVM off-heap = 2000MB - 1000MB = 1000MB
2,JVM Metaspace = 256MB
3,First, according to JVM overhead  = (JVM overhead + 256M(Metaspace) + 2000(flink size)) * 0.1  
=> total process size =  2.448GB
=> JVM overhead = 2.448GB * 0.1 =  262843055B =  250.667MB(At 192 M～1GB)，As valid
4,Finalize total process size = 2.448GB && JVM overhead = 250.667MB

3 memory allocation for taskmanager

TaskManager is a node for computing user tasks. Rational memory allocation can make applications more stable. The fine-grained memory model of TaskManager is as follows.