Programmer Think

Big data, Spark, Hadoop, python Big data Hadoop installation and configuration 1, Hadoop pseudo distributed configuration 1. Create Hadoop user: sudo useradd -m hadoop -s /bin/bash # Create hadoop user sudo passwd hadoop # Change Password sudo adduser hadoop sudo # Add administrator privileges Log out and log in ...

Hadoop[03-03] access count test based on DFS and ZKFC (Hadoop 2.0) Prepare the environment Prepare multiple virtual machines and start dfs and zookeeper See link for details: Hadoop2.0 start DFS and Zookeeper Some data of multiple virtual machines are as follows numberhost nameHost domain nameip address①ToozkyToozky192.168.64.220②Toozky2T ...

Introduction: MapReduce is a cluster based high-performance parallel computing platform. MapReduce is a software framework for parallel computing and operation. MapReduce is a parallel programming model and methodcharacteristic: ① The distribution is reliable. The operation of the data set is distributed to multiple nodes in the cluster to ac ...

kafka installation record: Official website: http://kafka.apache.org/downloads.html 1. Download yum install -y wget wget https://mirrors.bfsu.edu.cn/apache/kafka/2.6.0/kafka_2.12-2.6.0.tgz 2. Decompression: tar -zxvf kafka_2.12-2.6.0.tgz -C /opt/ 3. Change of name mv kafka_2.12-2.6.0 kafka 4. Create files cd kafka Create un ...

There is a requirement in the development: after creating ldap users, they need to be automatically synchronized to hue After viewing the official documents, you can use http to request the interface of hue and synchronize ldap users Record the pit of this interface call and the solution 1, hue's landing The related interface requests of hue ...

Original article publishing address: https://www.jianshu.com/p/d0e8b605c4ce Click "read the original text" or visit https://linkis.apache.org/#/ Learn more about Apache links Community developer: Maria Carrie GitHub : mindflow94   This article is mainly used to guide users to install and deploy Linkis and datasphere studio, and t ...

Mslab (memory local allocation buffer) of HBase preface This paper briefly introduces the write cache MemStore and data structure of HBase, as well as the function and source code analysis of MSLAB, which is the main component of write cache. MSLAB is the abbreviation of MemStore local allocation buffer. It carries out reasonable planning ...

background With the continuous expansion of data volume, the strengthening of analysis demand and the process of localization, the original traditional database can no longer support and complete heavy tasks. More and more data warehouses and data Lake platforms are migrating to MPP, big data and cloud original platforms. However, in thi ...

HADOOP installation Linux stand-alone Download Hadoop Hadoop3.xx download address: http://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/ Upload to Linux via FTP Decompression software tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/ Configure HADOOP environment variables Create custom profile vim /etc/prof ...

preface Part of the content is extracted from the training materials of Shang Silicon Valley, dark horse and so on 1. Get to know MapReduce 1.1 understand MapReduce idea   MapReduce thought can be seen everywhere in life, and everyone has been exposed to it more or less. The core idea of MapReduce is "divide and then combine ...