Presto cluster deployment (upgrade)

Posted by B of W on Tue, 11 Feb 2020 11:48:16 +0100

Let's talk about the last step of preparing to upgrade presto, because the pre deployed Presto machine itself deployed a CDH cluster (customer's home), and the machine was Centos6.x. according to the JDK on the Cloudrea official website, it can't move. Later, it was found that his JDK environment variable was written in / etc/profile, that is, he wanted to write a new PATH under the user with the priority of the user's own PATH - this was ok, of course, because he forgot to write $ A handful of bitter tears

The address of the old package first: link: https://pan.baidu.com/s/1U676CujPaD8kfjrkXgj0qQ
Extraction code: zvy0

1, Environmental preparation
1. Service role distribution

IP address Duty
172.16.180.4 coordinator
172.16.180.17 worker
172.16.180.3 worker
172.16.180.3 worker

2. installation package

Need to get it at the beginning of Baidu Online

Two, deployment
1. Install JDK
Do not go into details. It should be noted that the official website explains that presto-server-0.229.tar.gz requires JDK to be at least jdk-6u115. After field testing in the production environment (jdk-8u45-linux-x64.gz), this version is not available. Later, it is OK to change this version (jdk-8u221-linux-x64.tar.gz). The first deployed environment variable is best written to / etc/profile. If there is j DK, the environment variable is written to the corresponding user's.bashrc

2. Install presto
1) Create a new Presto user and upload presto-server-0.229.tar.gz to / home/presto /

# useradd presto
# chown -R presto. presto /home/presto
# su – presto
$ tar -zxvf presto-server-0.229.tar.gz

2) Modify configuration file (operation on 172.16.180.12)

# su – presto
$ cd /home/presto/presto-server-0.229
$ mkdir etc
$ cd etc/
$ touch node.properties
$ touch jvm.config
$ touch config.properties
$ touch log.properties
$ mkdir catalog
$ cd catalog
$ touch hive.properties
$ mkdir hive ##Upload hive's configuration files core-site.xml and hdfs-site.xml to this file
$ vi hive.properties
	connector.name=hive-hadoop2
	hive.metastore.uri=thrift://172.16.180.4:9083
	hive.config.resources=/home/presto/presto-server-0.229/etc/catalog/hive/core-site.xml,/home/presto/presto-server-0.229/etc/catalog/hive/hdfs-site.xml
:wq Save exit
$ cd ..
$ vi config.properties
coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=19999
query.max-memory=8GB
query.max-memory-per-node=2GB
discovery-server.enabled=true
discovery.uri=http://172.16.180.12:19999
:wq Save exit
$ vi jvm.config
-server
-Xmx8G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
:wq Save exit
$ vi log.properties
com.facebook.presto=INFO #Output log level
:wq Save exit
$ vi node.properties
node.environment=production
node.id=cgn_presto_coordinator_node1 #Node unique identification
node.data-dir=/home/presto/presto-server-0.229/data
:wq Save exit
$ exit
# scp -r /home/presto/presto-server-0.229 root@172.16.180.17: /home/presto/
# scp -r /home/presto/presto-server-0.229 root@172.16.180.3: /home/presto/

Modify config.properties and node.properties on 172.16.180.17 and 172.16.180.3, where config.properties are the same as the following:

coordinator=false
http-server.http.port=19999
query.max-memory=8GB
query.max-memory-per-node=2GB
discovery.uri=http://172.16.180.12:19999

On 172.16.180.17, the node.properties are as follows:

node.environment=production
node.id=cgn_presto_coordinator_node2
node.data-dir=/home/presto/presto-server-0.229/data

On 172.16.180.3, the node.properties are as follows:

node.environment=production
node.id=cgn_presto_coordinator_node3
node.data-dir=/home/presto/presto-server-0.229/data
  1. Start presto on 172.16.180.12, 172.16.180.17, 172.16.180.3
$ cd /home/presto/presto-server-0.229/bin
$ ./launcher start


If the above information appears, the startup is successful.
Browser login: http://172.16.180.12:19999

4) Upload presto-cli-0.229-executable.jar to / home/presto/presto-server-0.229/bin directory (connect to hive data warehouse)

$ cd /home/presto/presto-server-0.229/bin
$ mv presto-cli-0.229-executable.jar presto
$ chmod +x presto
$ vi hive.sh 
./presto --server 172.16.180.12:19999 --catalog hive --schema test
:wq Save exit
$ ./hive.sh


Connection succeeded!
3, Error resolution
1) error 1
Solution: add this parameter to config.properties: query.max-total-memory-per-node=2GB(2GB value as reference)
2) If this error occurs: (injection constructor error, java.lang. Invalid memory configuration. The sum of the maximum query total memory (2147483648) and the heap head space (644245094) of each node cannot be greater than the available heap memory (2147483648)

Solution: modify the - Xmx2G parameter of etc/jvm.config file to adjust the size

Published 10 original articles, won 0 praise and 361 visitors
Private letter follow

Topics: hive JDK xml jvm