Programmer Think

Product dimension data loading (zipper table) Zipper watch design: 1. Collect the full data of the day and store it in the ND (current day) table. 2. You can take out yesterday's full data from the history table and store it in the OD (last day's data) table. 3. ND-OD is the data added and changed on ...

Article directory 1. Cache table data in memory 2. Parameter optimization 1. Cache table data in memory Performance tuning is mainly about putting data into memory. Caching data in memory can improve performance by directly reading the value of memory. In RDD, use rdd.cache or rdd.persist to cac ...

1. architecture 2.Flume configuration Create a new configuration file under $flume \ home / conf: flume \ push \ streaming.conf The configuration idea is as follows: source select netcat and configure the host name and port Select avro for sink, and configure the host name and port channel s ...

Spark+Kafka Build Real-Time Analysis Dashboard I. Framework Spark+Kafka is used to analyze the number of male and female students shopping per second in real time, Spark Streaming is used to process the user shopping log in real time, then websocket is used to push the data to the browser in real ti ...

The Spark mentioned here includes SparkCore/SparkSQL/SparkStreaming. In fact, all operations are the same. The following shows the code in the actual project. Method 1: write the entire DataFrame to MySQL at one time (the Schema of DataFrame should be consistent with the domain name defined in the MySQL table) Dat ...

(1) case of ashes -- UDTF seeking wordcount Data format:Each line is a string and separated by spaces.Code implementation: object SparkSqlTest { def main(args: Array[String]): Unit = { //Block redundant logs Logger.getLogger("org.apache.hadoop").setLevel(Level.WARN) Logger.getLogger("org.apache.spark").setLevel(Leve ...

Download Spark on Spark official website Spark Download The version is free. After downloading, extract it and put it under bigdata (the directory can be changed) Download the file winutils.exe required by Hadoop under Windows Let's find it on the Internet. It won't be uploaded here. In fact, this file is optional, and error reporting doesn' ...

Introduction case: object SparkSqlTest { def main(args: Array[String]): Unit = { //Block redundant logs Logger.getLogger("org.apache.hadoop").setLevel(Level.WARN) Logger.getLogger("org.apache.spark").setLevel(Level.WARN) Logger.getLogger("org.project-spark").setLevel(Level.WARN) //Building programmin ...

JDBC data source Spark SQL supports reading data from relational databases (such as MySQL) using JDBC. The read data, still represented by DataFrame, can be easily processed using various operators provided by Spark Core. Created by: To connect Mysql during query:   It is very useful to use Spark SQL to process dat ...

Software version:     jdk: 1.8     maven: 3.61    http://maven.apache.org/download.cgi     spark: 2.42      https://archive.apache.org/dist/spark/spark-2.4.2/ Hadoop version: hadoop-2.6.0-cdh5.7.0 (Hadoop version supported by spark compilation, does not need to be installed) To configure maven: #Configure environment variables [root@hadoop004  ...