background

On October 4, 2018, two big data pioneers cloudera and Hortonworks announced the equal merger. Cloudera acquired Hortonworks in the form of shares, and cloudera shareholders finally obtained 60% of the shares of the merged company.
In November 2020, Cloudera announced that from January 31, 2021, all Cloudera software needs a valid subscription for access. This includes all previous versions of Cloudera distribution (CDH), Hortonworks Data Platform (HDP), Data Flow (HDF / CDF), and Cloudera Data Science Workbench (CDSW) with Apache Hadoop.

content validity

This series will be divided into three parts:

1. Xunlei YYDS (deploy Ambari local image warehouse)
2. Install ambari server and ambari agent
3. Deploy Ambari big data cluster

Download resources

Due to the blockade, we can't download the required resources. After trying various methods, the author thinks of a forgotten tool - Xunlei.

After opening a member, you can directly use the official website to connect and download the required resources. I think someone downloaded the corresponding resource before, and Xunlei cached it in its server. I uploaded these resources to Baidu online disk and Alibaba cloud disk, and found that they were almost transmitted in seconds. So you know!
The corresponding download link, I will put it in resources.

Build environment

Virtual machine: Oracle VM VirtualBox

Operating system image: CentOS-7-x86_64-Minimal-2009.torrent.iso

Three virtual machines were created, all of which were allocated 2 cores and 4GB of memory

Skip the operating system installation and SSH startup configuration section here.

Now let's officially enter the part of deploying Ambari image warehouse.

Deploy Ambari local mirror warehouse

Security configuration

Permanently open port 80

firewall-cmd --zone=public --add-port=80/tcp --permanent

service iptables restart

firewall-cmd --reload

Close SELinux

vi /etc/selinux/config

Change the value of SELINUX to = disabled, and then

Here, we also need to configure port mapping for VirtualBox

Installing Nginx

Install Yum utils

yum install -y yum-utils

Configure the YUM source for Nginx

vi /etc/yum.repos.d/nginx.repo

The contents are as follows

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

install Nginx
```bash
yum install -y nginx

Start Nginx

systemctl start nginx

Use the browser to access localhost. When the content appears, it indicates that nginx has been successfully installed!

Set nginx startup self startup

systemctl enable nginx

Configure mirror warehouse

Serious people who use the default profile? So:

mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.bak

It's a good habit to back up rather than delete!

Create ambari conf

The contents are as follows

Create directory

mkdir -p /var/www/html

Upload the following resources to / var/www/html

• ambari-2.7.5.0-centos7.tar.gz
• HDP-3.1.5.6091-centos7-rpm.tar.gz
• HDF-3.4.1.1-centos7-rpm.tar.gz
• HDP-GPL-3.1.5.0-centos7-gpl.tar.gz
• hdf-ambari-mpack-3.4.1.1-4.tar.gz
• HDP-UTILS-1.1.0.22-centos7.tar.gz

Unzip ambari-2.7 5.0-centos7. tar. GZ to / var/www/html

tar xf ambari-2.7.5.0-centos7.tar.gz -C /var/www/html

Create hdf and hdp folders

mkdir -p /var/www/html/hdf && mkdir -p /var/www/html/hdp

Unzip the relevant files to the corresponding folder

tar xf HDF-3.4.1.1-centos7-rpm.tar.gz -C /var/www/html/hdf && tar xf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp && tar xf HDP-GPL-3.1.5.0-centos7-gpl.tar.gz -C /var/www/html/hdp

Restart Nginx

systemctl restart nginx

"Index of /" appears in the upper left corner, indicating that the configuration is successful.

So far, a local image warehouse has been deployed.

reference material

Access to Cloudera software
nginx: Linux packages
Accessing HDP Repositories
HDF Repository Locations