Hadoop 2.x.y (pseudo distributed)

Refer to the single node setup section of the corresponding version of the official website
https://hadoop.apache.org/docs/

First, ssh and rsync

Then download the bin package and extract it. Add the extracted root directory as the environment variable Hadoop? Home

# example
export HADOOP_HOME=/home/yzj/Applications/hadoop/hadoop-2.10.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Modify Hadoop? Home / etc / Hadoop / hadoop-env.sh

# set to the root of your Java installation
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
# The Java path can be found by the whereis java command

Modify Hadoop? Home / etc / Hadoop / core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Modify Hadoop? Home / etc / Hadoop / hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/yzj/Applications/hadoop/hadoop-2.10.0/tmp</value>
        <!-- By default, it will be stored in linux Systematic /tmp Must be modified to a persistent path -->
    </property>
</configuration>

Set ssh

Start ssh service: sudo systemctl start sshd.service
Let's see if we can ssh to localhost: ssh localhost without entering the password
No way.

ssh-keygen -t rsa -P '' -f ~/.ssh/id\_rsa
cat ~/.ssh/id\_rsa.pub >> ~/.ssh/authorized\_keys
chmod 0600 ~/.ssh/authorized\_keys

Modify Hadoop? Home / etc / Hadoop / mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

Modify Hadoop? Home / etc / Hadoop / yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce\_shuffle</value>
    </property>
</configuration>

Format hdfs

$HADOOP_HOME/bin/hdfs namenode -format

Run hdfs

$HADOOP_HOME/sbin/start-dfs.sh

Create user folder

$HADOOP_HOME/bin/hdfs dfs -mkdir /user

Create user folder

$HADOOP_HOME/bin/hdfs dfs -mkdir /user/<username>
# Corresponding to the current user of linux eg. hdfs dfs -mkdir /user/yzj

Run yarn

$HADOOP_HOME/sbin/start-yarn.sh

Visit hdfs under localhost:50070 and yarn under localhost:8088

Sign out

$HADOOP_HOME/sbin/stop-yarn.sh

MariaDB

https://wiki.archlinux.org/index.php/MariaDB

Installing MariaDB

sudo pacman -S mariadb

After installation

sudo mysql\_install\_db --user=mysql --basedir=/usr --datadir=/var/lib/mysq

Open mysql service

sudo systemctl start mariadb.service

implement

sudo mysql\_secure\_installation

During the process, you need to set the password for the root user and keep the test database

Log in mysql as root

sudo mysql -u root -p

Switch from none to mysql database after login

use mysql;

Update authority

update user set host = '%' where user = 'root'

Refresh

flush privileges;

Sign out

quit;

Hive 2.x.y

https://blog.csdn.net/qq_37106028/article/details/78247727

Download and extract bin package, set environment variables

# example
export HIVE_HOME=/home/yzj/Applications/hadoop/hive-2.3.6
export PATH=$PATH:$HIVE_HOME/bin

create profile

cp $HIVE_HOME/conf/hive-env.sh.template hive-env.sh
cp $HIVE_HOME/conf/hive-default.xml.template hive-site.xml

Modify hiv-env.sh

JAVA_HOME=/usr/lib/jvm/java-8-openjdk
HADOOP_HOME=$HADOOP_HOME
HIVE_HOME=$HIVE_HOME

Modify hive-site.xml

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>org.mariadb.jdbc.Driver</value><!-- No com.mysql.jdbc.Driver -->
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>Self modification</value>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>Set up a persistent directory</value>
    <!-- eg. /home/yzj/Applications/hadoop/hive-2.3.6/tmp -->
</property>
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>Ditto</value>
</property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>Ditto</value>
</property>

Copy and drive to HIVE_HOME/lib directory (use mariadb-java-client-x.y.z.jar instead of mysql-connector-java-x.y.z.jar)
Download address https://mariadb.com/downloads/#connectors

Delete the jline jar package in the directory of Hadoop ﹣ home / share / Hadoop / yarn / lib, and replace it with the jline jar package in the directory of hive ﹣ home / lib (this step can be omitted if there is no jline jar package in the directory corresponding to Hadoop)

Start hive (start Hadoop first)

$HIVE_HOME/bin/hive