Presto memory principle (outdated, but you can refer to it)

Posted by thegreatone2176 on Sat, 26 Feb 2022 08:18:41 +0100

Presto Is an open source distributed SQL query engine, which is suitable for interactive analysis and query. The amount of data supports GB to PB bytes. Presto supports online data query, including Hive, Cassandra, relational database and proprietary data storage. A Presto query can merge data from multiple data sources and analyze across the entire organization. Presto targets the needs of analysts, who expect a response time of less than 1 second to a few minutes. Presto ended the dilemma of data analysis, either using fast and expensive business solutions or using slow "free" solutions that consume a lot of hardware.

presto0.131 start to optimize the memory model until the online version 0.188 (including 0.161 of EMRV1) in the current EMRv2 uses this memory model. A mechanism called memory pool is used to manage the memory usage of tasks in Presto and presto itself.

Design ideas and goals can be viewed: https://github.com/prestodb/presto/issues/2624

presto has achieved its goals:

It is easier for users to control the memory limit of each query.
Prevent memory overflow and resulting crashes.
Make full use of the memory of the cluster without being limited by the memory resources of the whole cluster caused by a "big query".

Presto memory tuning parameter

Presto divides the allocable memory (jvm Xmx) of each worker node into three parts: system memory pool, reserved memory pool and general memory pool. When Presto is started, they will be allocated with the initialization of the worker node, and then each worker node will be found through the service and reported to the coordinator node.

The following figure is the memory diagram of presto worker node:

Schematic diagram of memory distribution of worker} node

As can be seen from the schematic diagram, the memory heap size of a worker node can be divided into two parts at most: system reserved memory + query memory. The query memory is divided into maximum query memory + other query memory.

System reserved memory: the memory necessary for the initialization and execution of tasks of the worker node, including the regular reporting of the preto discovery service and the task management in each query data Structure, etc. Use resources The reserved system memory configuration item is configured. The default is 0.4 of the heap size of the worker node.

In addition to the memory reserved by the system, the rest memory for woker will be used for query:

Maximum query memory: the coordinator node will regularly schedule and view the duration and memory used by each query. In this process, it will find the query that consumes the most memory and schedule the maximum memory use for this query. This query can obtain the maximum amount of dedicated memory configured by each worker node. Using query The max memory per node configuration item can be configured. The default is 0.1 of the worker node heap size. This value can be set according to the peak Mem monitored by query as a reference.

Other query memory: in addition to the memory reserved by the system and the maximum query memory in the worker node heap, other query memory is used.

The heap memory configuration of the worker node is most related to the two scenarios used by users:

1. Amount / complexity of user query data

2. User query concurrency

1. Decide how much maximum query memory to use 2 Determines how much jvm heap to use.

For example, if a batch of (n=5) queries are submitted to a presto cluster with 10 worker nodes at the same time, and the query statement with the highest data volume / complexity occupies 20GB of memory in one node, we should give each worker node the maximum query memory, that is, 20GB/10 = 2GB, which needs to be executed concurrently, Then the query memory should be 2GB*5, about 10GB. At this time, it can be deduced that each worker node should be configured with a heap size of 10 / 0.6 = 17. (0.4 is reserved temporarily, and 0.6 is reserved for query).

In addition to the above three configurations, there is another configuration that needs attention, that is, the maximum supported memory of query: it means the maximum supported memory of each query. The configuration item is: query Max memory can be configured as query max-memory-per- node *The number of worker s in the above example is 2GB*10=20GB.

Using these parameters can basically solve most of the slow query and OOM problems encountered when using presto cluster. Of course, more detailed memory management needs to be done for a presto cluster: for example, memory scheduling for users, such as using queuing to restrict access to the cluster and limiting the memory limit used by the query of the whole cluster, such as the memory fine management of the coordinator. You can view the next article presto tuning intermediate article.

Presto memory tuning principle

After reading the previous part, you can intuitively distribute the console operation in emr configuration and practice it. You can continue to look down for understanding the principle and troubleshooting deeper reasons (start talking about the principle from the perspective of source code, because the source code can understand all the details):

public final class LocalMemoryManager{
     
    public static final MemoryPoolId GENERAL_POOL = new MemoryPoolId("general");
    public static final MemoryPoolId RESERVED_POOL = new MemoryPoolId("reserved");
    public static final MemoryPoolId SYSTEM_POOL = new MemoryPoolId("system");
<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">private</span> final DataSize maxMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">private</span> final Map<span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&lt;</span>MemoryPoolId<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> MemoryPool<span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&gt;</span> pools<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>

@Inject    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">public</span> <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">LocalMemoryManager</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>NodeMemoryConfig config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> ReservedSystemMemoryConfig systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">requireNonNull</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token string" style="list-style:inherit;color:rgb(102,153,0);">"config is null"</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">requireNonNull</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token string" style="list-style:inherit;color:rgb(102,153,0);">"systemMemoryConfig is null"</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    long maxHeap <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> Runtime<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getRuntime</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">maxMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">checkArgument</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&lt;</span> maxHeap<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token string" style="list-style:inherit;color:rgb(102,153,0);">"Reserved memory %s is greater than available heap %s"</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">DataSize</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>maxHeap<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> BYTE<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    maxMemory <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">DataSize</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>maxHeap <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">-</span> systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> BYTE<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>

    ImmutableMap<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>Builder<span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&lt;</span>MemoryPoolId<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> MemoryPool<span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&gt;</span> builder <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> ImmutableMap<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">builder</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">checkArgument</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&lt;=</span> maxMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">format</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token string" style="list-style:inherit;color:rgb(102,153,0);">"%s set to %s, but only %s of useable heap available"</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> QUERY_MAX_MEMORY_PER_NODE_CONFIG<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> maxMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    builder<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">put</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>RESERVED_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">MemoryPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>RESERVED_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    DataSize generalPoolSize <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">DataSize</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>Math<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">max</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token number" style="list-style:inherit;color:rgb(153,0,85);">0</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> maxMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">-</span> config<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">toBytes</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> BYTE<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    builder<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">put</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>GENERAL_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">MemoryPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>GENERAL_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> generalPoolSize<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    builder<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">put</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>SYSTEM_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">MemoryPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>SYSTEM_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> systemMemoryConfig<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">this</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>pools <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> builder<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">build</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span><div class="hide-preCode-box"><span class="hide-preCode-bt"><img class="look-more-preCode contentImg-no-view" src="https://csdnimg.cn/release/blogv2/dist/pc/img/newCodeMoreWhite.png" alt="" title=""></span></div></pre>

The size of the reservedmemorypool is based on config The properties file configures query Max memory per node decision. The default value of this value is 0.1 of jvm(xmX).

public class NodeMemoryConfig{
  
    public static final String QUERY_MAX_MEMORY_PER_NODE_CONFIG = "query.max-memory-per-node";
<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">private</span> DataSize maxQueryMemoryPerNode <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">DataSize</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>Runtime<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getRuntime</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">maxMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">*</span> <span class="token number" style="list-style:inherit;color:rgb(153,0,85);">0.1</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> BYTE<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>

@NotNull    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">public</span> DataSize <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">return</span> maxQueryMemoryPerNode<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span>

@<span class="token function" style="list-style:inherit;color:rgb(221,74,104);">Config</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>QUERY_MAX_MEMORY_PER_NODE_CONFIG<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">public</span> NodeMemoryConfig <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">setMaxQueryMemoryPerNode</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>DataSize maxQueryMemoryPerNode<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">this</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>maxQueryMemoryPerNode <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> maxQueryMemoryPerNode<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">return</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">this</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span></pre>

The size of systemMemoryPool depends on the configured resources Determined by reserved system memory, the default value of this value is 0.4 of jvm(xmX).

public class ReservedSystemMemoryConfig{
  
    private DataSize reservedSystemMemory = new DataSize(Runtime.getRuntime().maxMemory() * 0.4, BYTE);
@NotNull    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">public</span> DataSize <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">return</span> reservedSystemMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span>

@<span class="token function" style="list-style:inherit;color:rgb(221,74,104);">Config</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token string" style="list-style:inherit;color:rgb(102,153,0);">"resources.reserved-system-memory"</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">public</span> ReservedSystemMemoryConfig <span class="token function" style="list-style:inherit;color:rgb(221,74,104);">setReservedSystemMemory</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>DataSize reservedSystemMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">this</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>reservedSystemMemory <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> reservedSystemMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">return</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">this</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span></pre>

Maximum memory scheduling policy

Presto is a query statement, which is transformed into a logical execution plan based on stage. Then, the logical execution plan is transformed into a physical execution plan, which is divided into parallel tasks among worker nodes, and each task is transformed into a specific Operator for actual execution. Before the actual implementation of the physical plan, the memory requirements come from the systemMemoryPool, including temporary data structure, transmission buffer, etc. When executing a physical plan, different types of operators apply for memory as needed. For example, the aggregationOperator uses the getestimatedsize () method to estimate the required memory. The memory obtained here comes from the reservedmemorypool or generalMemoryPool. Which pool to use depends on whether the current query consumes the most memory.

When creating a query, generalMemoryPool is used by default:

public SqlTaskManager(
            LocalExecutionPlanner planner,
            LocationFactory locationFactory,
            TaskExecutor taskExecutor,
            QueryMonitor queryMonitor,
            NodeInfo nodeInfo,
            LocalMemoryManager localMemoryManager,
            TaskManagementExecutor taskManagementExecutor,
            TaskManagerConfig config,
            NodeMemoryConfig nodeMemoryConfig,
            LocalSpillManager localSpillManager,
            NodeSpillConfig nodeSpillConfig)
    {
       ...

queryContexts = CacheBuilder.newBuilder().weakValues().build(CacheLoader.from(

            queryId <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">-</span><span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&gt;</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">QueryContext</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>
                    queryId<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    maxQueryMemoryPerNode<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    localMemoryManager<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>LocalMemoryManager<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>GENERAL_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    localMemoryManager<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>LocalMemoryManager<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span>SYSTEM_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    taskNotificationExecutor<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    driverYieldExecutor<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    maxQuerySpillPerNode<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span>
                    localSpillManager<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getSpillSpaceTracker</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span></pre>

presto will regularly check the memory consumed by all queries. This timer is constructed during presto initialization. The implementation is as follows:

queryManagementExecutor.scheduleWithFixedDelay(() -> {
  
    ...
    enforceMemoryLimits();
    ...}, 1, 1, TimeUnit.SECONDS);

The updateAssignments method is used to find the most memory consuming query and put it into RESERVED_POOL:

if (reservedPool.getAssignedQueries() == 0 && generalPool.getBlockedNodes() > 0) {
  
    QueryExecution biggestQuery = null;
    long maxMemory = -1;
    for (QueryExecution queryExecution : queries) {
  
    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">if</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">resourceOvercommit</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>queryExecution<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getSession</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>     
        <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">continue</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span>

    long bytesUsed <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> queryExecution<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">getTotalMemoryReservation</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>

    <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">if</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>bytesUsed <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">&gt;</span> maxMemory<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
        biggestQuery <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> queryExecution<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
        maxMemory <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">=</span> bytesUsed<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
    <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span>

<span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">if</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>biggestQuery <span class="token operator" style="list-style:inherit;color:rgb(166,127,89);background:rgba(255,255,255,.5);">!=</span> <span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">null</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span> <span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">{<!-- --></span>
    biggestQuery<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">.</span><span class="token function" style="list-style:inherit;color:rgb(221,74,104);">setMemoryPool</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span><span class="token keyword" style="list-style:inherit;color:rgb(0,119,170);">new</span> <span class="token class-name" style="list-style:inherit;">VersionedMemoryPoolId</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">(</span>RESERVED_POOL<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">,</span> version<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">)</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">;</span>
<span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span><span class="token punctuation" style="list-style:inherit;color:rgb(153,153,153);">}</span><div class="hide-preCode-box"><span class="hide-preCode-bt"><img class="look-more-preCode contentImg-no-view" src="https://csdnimg.cn/release/blogv2/dist/pc/img/newCodeMoreWhite.png" alt="" title=""></span></div></pre>

Looking at the updateNodes method in the outer layer, we can find that this allocation strategy of RESERVED POOL will be applied to each node. In other words, this query will monopolize the space of RESERVED POOL at each node.

for (RemoteNodeMemory node : nodes.values()) {
  
    node.asyncRefresh(assignments);}

Presto memory monitoring

You can use the webui provided by presto to instantly track the memory usage of the whole cluster or each query execution. presto's webui uses DI framework Guice in the background and jquery+react in the foreground.

presto cluster overview

presto a query monitoring chart

Reservedmemory in a web page is the size of the ReservedMemoryPool. In the query resource summary, peakMemory indicates the peak value of memory used by the current query. memory pool indicates the currently used pool(reserved or generic).

The official account recommends:

Topics: presto

Programmer Think

Presto memory principle (outdated, but you can refer to it)

Presto memory tuning parameter

Presto memory tuning principle

Maximum memory scheduling policy

Presto memory monitoring

Hot Topics