Programmer Think

Exception code description Just beginning to contact Hadoop，about MapReduce From time to time, I especially understand that the following records the problems and solutions that have been tangled for a day 1. Execute MapReduce task hadoop jar wc.jar hejie.zheng.mapreduce.wordcount2.WordCountDriver /input /output 2. Jump out of exception ...

Pseudo distributed clusters are shown in: https://blog.csdn.net/weixin_40612128/article/details/119008295?spm=1001.2014.3001.5501 After the pseudo distributed cluster is completed, let's take a look at what the real distributed cluster is like. Take a look at this figure. It shows three nodes. The one on the left is the master node and the t ...

9.1 migrate the data of the original CM node to the new node 9.1.1 backup the original CM node data It mainly backs up the monitoring data and management information of CM. The data directory includes: /var/lib/cloudera-host-monitor /var/lib/cloudera-service-monitor /var/lib/cloudera-scm-server /var/lib/cloudera-scm-eventserver /var/lib/clou ...

1, Analysis target 1. Establish daily operation index system; 2. Analyze the composition of existing users and count the composition of daily users; 3. Analyze the re purchase situation of users and guide the subsequent re purchase operation. 2, Data description User_info user information table: user_action user behavior table: 3, Imp ...

3.2 RDD programming In Spark, RDD is represented as an object, and RDD is converted through method calls on the object. After defining RDD through a series of transformations, you can call actions to trigger the calculation of RDD. Actions can be to return results to the application (count, collect, etc.) or to save data to the storage system ...

Mapper class // // Source code recreated from a .class file by IntelliJ IDEA // (powered by Fernflower decompiler) // package org.apache.hadoop.mapreduce; import java.io.IOException; import org.apache.hadoop.classification.InterfaceAudience.Public; import org.apache.hadoop.classification.InterfaceStability.Stable; @Public @Stable public cla ...

11.1 introduction to experimental environment The cluster environment is running normallyHive and Impala services are installed in the clusterOperating system: redhat6 fiveCDH and CM versions are 5.11.1EC2 user with sudo permission is used for operation 11.2 UDF function development - using Intellij tools Use Intellij tool to develop Hive's ...

1. Introduction to big data 1.1 big data concept big data refers to a data set that cannot be captured, managed and processed by conventional software tools within a certain time range. It is a massive, high growth rate and diversified information asset that requires a new processing mode to have stronger decision-making power, insight an ...

Shared variable broadcast variable Broadcast variables allow programs to cache a read-only variable on each machine in the cluster instead of saving a copy for each task. With broadcast variables, you can share some data in a more efficient way, such as a global configuration file. from pyspark.sql import SparkSession spark = SparkSe ...

zeppelin installation configuration Installation premise: before installing zeppelin, make sure your hadoop and hive are on. Oh, this is very important!!! First step Put the downloaded compressed package under / opt/download/hadoop directory (I put it here, and you can choose by yourself) cd /opt/download/hadoop ls Then just drag it Step ...