Programmer Think

background With the continuous expansion of data volume, the strengthening of analysis demand and the process of localization, the original traditional database can no longer support and complete heavy tasks. More and more data warehouses and data Lake platforms are migrating to MPP, big data and cloud original platforms. However, in thi ...

HADOOP installation Linux stand-alone Download Hadoop Hadoop3.xx download address: http://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/ Upload to Linux via FTP Decompression software tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/ Configure HADOOP environment variables Create custom profile vim /etc/prof ...

Pseudo distributed clusters are shown in: https://blog.csdn.net/weixin_40612128/article/details/119008295?spm=1001.2014.3001.5501 After the pseudo distributed cluster is completed, let's take a look at what the real distributed cluster is like. Take a look at this figure. It shows three nodes. The one on the left is the master node and the t ...

Xiaobai's big data journey (73) Flume advanced Last review The previous chapter introduced the internal principle of Flume. This chapter explains the extended knowledge of Flume. The focus of this chapter is to understand and learn to use the user-defined components of Flume Custom components The internal principle was introduced in the pr ...

preface As we all know, HDFS has three sets to ensure the high availability of its data. Moreover, the placement of HDFS on the three replicas is also carefully designed. Two replicas are placed on the same rack (different nodes), and the other replica is placed on another rack. Under such a placement strategy, the replica data can toler ...

Environmental description and purpose Preparation: I prepared three virtual machines myself Using Hyper-V to build virtual machine cluster environment on windows platform_ a18792721831 blog - CSDN blog The environment is as follows hostnameNodedataNoderesourceManagernodeManagerOpening to the outside worldhadoop01start-upDo not startstart ...

First, build haoop2.0 of eclipse 7.1 development environment, the resources used are linked as follows: Install Hadoop 2.0 for windows 7.1 environment Building hadoop development environment under eclipse In this way, we can develop hadoop in eclipse catalogue 1, Introduction to MapReduce model 1. Map and Reduce functions 2. MapReduce a ...

1. Install Centos system The software list is as follows: WMware Workstation 12 CentOS-7-x86_64-DVD-1810.iso jdk-8u181-linux-x64.tar.gz hadoop-2.7.3.tar.gz hbase-1.2.7-bin.tar.gz apache-hive-2.1.1-bin.tar.gz kafka_2.11-1.1.0.tgz spark-2.3.2-bin-hadoop2.7.tgz apache-flume-1.8.0-bin.tar.gz Note: since three hosts need to be simulated on the ...

reference material Video data Operating environment Windows10JDK8IDEA 2021.6 professionalHadoop3.1.3CentOS7Three Hadoop fully distributed cluster nodes 1, Preparing the HDFS Java API environment 1.1 preparing Hadoop environment in windows system Hadoop3.1.3 official download address: Click download , download and unzip. hadoop ...

1. DataX template Method 1: DataX configuration file template python bin/datax.py -r mysqlreader -w hdfswriter Method 2: Official Documents https://github.com/alibaba/DataX/blob/master/README.md 2. Synchronize Mysql data to HDFS cases 2.1 TableMode of mysqlreader Use attributes such as table, column, and where to declare the data t ...