OushuDB implements the use range from DBLINK to ORACLE

background With the continuous expansion of data volume, the strengthening of analysis demand and the process of localization, the original traditional database can no longer support and complete heavy tasks. More and more data warehouses and data Lake platforms are migrating to MPP, big data and cloud original platforms. However, in thi ...

Posted by manishdugar on Mon, 07 Feb 2022 21:57:33 +0100

Hadoop installation complete

HADOOP installation Linux stand-alone Download Hadoop Hadoop3.xx download address: http://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/ Upload to Linux via FTP Decompression software tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/ Configure HADOOP environment variables Create custom profile vim /etc/prof ...

Posted by cesarcesar on Mon, 07 Feb 2022 09:17:24 +0100

Hadoop distributed cluster installation

Pseudo distributed clusters are shown in: https://blog.csdn.net/weixin_40612128/article/details/119008295?spm=1001.2014.3001.5501 After the pseudo distributed cluster is completed, let's take a look at what the real distributed cluster is like. Take a look at this figure. It shows three nodes. The one on the left is the master node and the t ...

Posted by verlen on Wed, 02 Feb 2022 19:57:20 +0100

Big data journey for beginners who play strange and upgrade < Flume advanced >

Xiaobai's big data journey (73) Flume advanced Last review The previous chapter introduced the internal principle of Flume. This chapter explains the extended knowledge of Flume. The focus of this chapter is to understand and learn to use the user-defined components of Flume Custom components The internal principle was introduced in the pr ...

Posted by xgab on Tue, 25 Jan 2022 12:08:08 +0100

Design and implementation of block placement policy for HDFS multi rack distribution

preface As we all know, HDFS has three sets to ensure the high availability of its data. Moreover, the placement of HDFS on the three replicas is also carefully designed. Two replicas are placed on the same rack (different nodes), and the other replica is placed on another rack. Under such a placement strategy, the replica data can toler ...

Posted by polymnia on Sun, 23 Jan 2022 08:42:49 +0100

hadoop cluster construction and hadoop configuration

Environmental description and purpose Preparation: I prepared three virtual machines myself Using Hyper-V to build virtual machine cluster environment on windows platform_ a18792721831 blog - CSDN blog The environment is as follows hostnameNodedataNoderesourceManagernodeManagerOpening to the outside worldhadoop01start-upDo not startstart ...

Posted by Ardivaba on Fri, 21 Jan 2022 22:06:41 +0100

Eclipse builds Hadoop environment and actual resource sharing

First, build haoop2.0 of eclipse 7.1 development environment, the resources used are linked as follows: Install Hadoop 2.0 for windows 7.1 environment Building hadoop development environment under eclipse In this way, we can develop hadoop in eclipse catalogue 1, Introduction to MapReduce model 1. Map and Reduce functions 2. MapReduce a ...

Posted by clown[NOR] on Fri, 21 Jan 2022 03:56:50 +0100

Big data cluster software installation manual 1

1. Install Centos system The software list is as follows: WMware Workstation 12 CentOS-7-x86_64-DVD-1810.iso jdk-8u181-linux-x64.tar.gz hadoop-2.7.3.tar.gz hbase-1.2.7-bin.tar.gz apache-hive-2.1.1-bin.tar.gz kafka_2.11-1.1.0.tgz spark-2.3.2-bin-hadoop2.7.tgz apache-flume-1.8.0-bin.tar.gz Note: since three hosts need to be simulated on the ...

Posted by snakez on Tue, 18 Jan 2022 09:25:26 +0100

Hadoop | HDFS learning notes | HDFS Java API environment construction | Java operating HDFS file system | multiple cases

reference material Video data Operating environment Windows10JDK8IDEA 2021.6 professionalHadoop3.1.3CentOS7Three Hadoop fully distributed cluster nodes 1, Preparing the HDFS Java API environment 1.1 preparing Hadoop environment in windows system Hadoop3.1.3 official download address: Click download , download and unzip. hadoop ...

Posted by TomT64 on Tue, 18 Jan 2022 07:33:14 +0100

Simple application of DataX

1. DataX template Method 1: DataX configuration file template python bin/datax.py -r mysqlreader -w hdfswriter Method 2: Official Documents https://github.com/alibaba/DataX/blob/master/README.md 2. Synchronize Mysql data to HDFS cases 2.1 TableMode of mysqlreader Use attributes such as table, column, and where to declare the data t ...

Posted by LonelyPixel on Tue, 11 Jan 2022 14:51:47 +0100