Flume collection 2-Flume introduction

I Flume installation deployment Installation address: Flume official website Address: http://flume.apache.org/ Document viewing address : http://flume.apache.org/FlumeUserGuide.html Download address : http://archive.apache.org/dist/flume/ Installation deployment: CDH 6.3 is used locally Version 1, Flume has been installed. The instal ...

Posted by safra on Sat, 25 Dec 2021 11:49:21 +0100

Introduction to spark - Spark operating environment

Reference link https://www.bilibili.com/video/BV11A411L7CK?p=11 Spark operating environment As a data processing framework and computing engine, Spark is designed to run in all common cluster environments. The mainstream environment in domestic work is yard, but the container environment is becoming more and more popular Local mode The s ...

Posted by FireyIce01 on Thu, 23 Dec 2021 14:01:46 +0100

National big data skills competition for college students (Hadoop cluster construction)

preface This blog will build Hadoop based on the data of the National College Students' big data skills competition in previous years and this year's training. There are corresponding screenshots of each step. The following blog is only used as a record of the process of building Hadoop ~ if there are deficiencies, you are welcome to point ...

Posted by johnsonzhang on Thu, 23 Dec 2021 09:25:26 +0100

Hadoop3. Upload file of X HDFS shell operation

1. copyFromLocal $ hadoop fs -help copyFromLocal -copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst> : Copy files from the local file system into fs. Copying fails if the file already exists, unless the -f flag is given. Flags: -p Preserves access and modification times, ow ...

Posted by lkalik on Wed, 22 Dec 2021 03:37:58 +0100

Three level practical subject report

Three level practical subject report Project Name: website log analysis system of canva paintable online graphic design software Major name: Data Science and big data technology Class: = = 201 Student No.:============ Student Name: CS daydream Instructor:=== – -- December 2021 – Abstract With the development of the Intern ...

Posted by rebelo on Sun, 19 Dec 2021 20:30:54 +0100

Yan web page port 8088 is not accessible in Windows browser

Problem description After configuring the Hadoop Namenode, secondarynode, and datanode nodes, configure mapred site in the hadoop /etc directory xml,yarn-site.xml file. After configuration, start resource manager and nodemanager. Among them, yen site The XML configuration is as follows: <property> <name>yarn.node ...

Posted by matbennett on Sun, 19 Dec 2021 13:56:35 +0100

Initial experience of Hadoop MapReduce operation

Initial experience of Hadoop MapReduce operation A simple MapReduce job requires a map function, a reduce function and some code to run the job package com.grits.hadoop.learning; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.ap ...

Posted by les4017 on Sat, 18 Dec 2021 17:57:06 +0100

Use of HBase ImportTSV tool

1. Importtsv function description Load text data in the format of tsv (or csv, each field in each row of data is separated by a separator) into the HBase table. 1) . load and import in Put mode 2) . bulk load and import are adopted Use the following command to view the instructions for the official HBase built-in tool class: HADOOP_HOME ...

Posted by margarette_a on Wed, 15 Dec 2021 21:19:27 +0100

Build a Hadoop cluster using three alicloud servers (collection of cloud computing experiments)

It's silly to use Alibaba cloud server to set up this hadoop cluster, which wastes me nearly a week. 1. Preparation Prepare three alicloud servers Namenode121.196.224.191Datanode1121.196.226.12Datanode247.96.42.166 Port setting (pit behind) Namenode Open 9000 ports manually through security groups View the private ip address of the Namenode ...

Posted by mort on Wed, 15 Dec 2021 18:59:18 +0100

HDFS transparent encryption usage, Keystore and Hadoop KMS, encryption area, key concepts and architecture of transparent encryption, KMS configuration

HDFS transparent encryption, Keystore and Hadoop KMS, encryption area The data in HDFS will be saved in the form of blocks in the local disk of each data node, but these blocks are in clear text. If you directly access the directory where the block is located under the operating system, you can directly view the contents through the cat comman ...

Posted by R0CKY on Mon, 13 Dec 2021 01:26:20 +0100