Programmer Think

Hive compression and storage 1, Hadoop compression configuration 1.1 MR supported compression coding Compression coding In order to support a variety of compression / decompression algorithms, Hadoop introduces an encoder / decoder, as shown in the following table: encoder Comparison of compression performance performance comparison ...

1. Using zipper table to implement DIM layer business dimension table in core transaction analysis and rollback of zipper table ODS Layer Data Preparation Because we need to construct our own data, we use the existing ods_trade_shops creates a new table: create database hw; drop table if exists hw.ods_trade_shops; create table hw.ods_trade_s ...

Hello, I'm Lao Wu. You can also call me classmate Wu. Younger friends can also call me senior brother Wu. Welcome to the world of data analysis with me and learn together! Interested friends can pay attention to me Data analysis column , there are many high-quality articles to share with you. In addition, you are also welcome to pay attention ...

1. hive basic data type 2. Data type Instance: create tables with different data types create table test( name string, friends array<string>, children map<string, int>, address struct<street:string, city:string> ) row format delimited fields terminated by ',' collection items terminated by '_' ma ...

Hive improved chapter Use of Hive Hive's bucket table 1. Principle of drum dividing table Bucket splitting is a more fine-grained partition relative to partition. Hive table or partition table can further divide bucketsDivide the bucket, take the hash value of the whole data content according to a column, and determine which bucket th ...

SQOOP installation and use SQOOP installation 1. Upload and unzip tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local/soft/ 2. Modify folder name mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop-1.4.7 3. Modify profile # Switch to the sqoop profile directory cd /usr/local/soft/sqoop-1.4.7/conf # Copy profile and rename cp sqoop-env ...

1, Introduction Hive has three types of UDFs: (normal) UDF, user-defined aggregate function (UDAF), and user-defined table generating function (UDTF). UDF: the operation acts on a single data row and produces a data row as output. Most functions, such as mathematical and string functions, fall into this category.UDAF: accepts multiple input d ...

catalogue The new architecture is integrated with the lake warehouse 1, Version Description 2, Compile and package Hudi version 0.10.0 1. Use git to clone the latest master on github 2. Compilation and packaging 3, Create a flick project 1. Main contents of POM document 2.checkpoint 3.flinkcdc code 4.hudi code (refer to the official ...

2021SC@SDUSC catalogue summary Supplementary description doPhase1() getMetaData(QB, ReadEntity) analysis summary In the last article, I analyzed doPhase1() Function, which is the initial stage of semantic analysis. The final goal of the program is to load the ast data into QB. The main idea of doPhase1 in this stage is to recursively tra ...

1, Hive installation 1. First, use xftp to transfer apache-hive-3.1.2-bin.tar.gz to the virtual machine, unzip it to / usr/local and change the file permissions. sudo tar -zxvf ./apache-hive-3.1.2-bin.tar.gz -C /usr/local # Unzip into / usr/local cd /usr/local/ sudo mv apache-hive-3.1.2-bin hive # Change the folder name to hive sudo c ...