Programmer Think

1. Error Key Information Caused by: org.postgresql.util.PSQLException: ERROR: column "c1" is of type point but expression is of type character at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285) at org.postgres ...

Hive related configuration is required for Spark sql to read and write hive, so hive site is generally used The XML file is placed in the conf directory of spark. Code calls are simple. The key is the source code analysis process and how spark interacts with hive. 1. Code call Read hive code SparkSession sparkSession = SparkSession.builder() ...

Spark SQL implementation principle - logic plan optimization - operation push down: EliminateOuterJoin rule This rule optimizes the outer join operation in order to eliminate the outer join operation as much as possible and convert it into inner or other join types. The EliminateOuterJoin optimization rule can take effect when the join operati ...

Spark adopts a large number of event listening methods to realize the communication between components on the driver side. This article will explain how to implement event listening in spark Observer mode and listener There is an observer pattern in the design pattern, which establishes a dependency between objects. When an object state chang ...

preface Spark computing framework encapsulates three data structures for high concurrency and high throughput data processing Handle different application scenarios. The three data structures are: RDD: distributed elastic datasetAccumulator: distributed shared write only variableBroadcast variables: distributed shared read-only variables C ...

Wang Hui, joined qunar.com in 2017. At present, he is responsible for anti crawler related risk control business, has a wide range of technical fields, and is continuously exploring the practice direction of intelligent risk control. I preface In Qunar intelligent risk control scenario, the risk control R & D team often applies som ...

What is Spark The whole Hadoop ecosystem is divided into distributed file system HDFS, computing framework MapReduce and resource scheduling framework Yan. However, with the development of the times, MapReduce's high-intensity disk IO, network communication frequency and dead write make it seriously slow down the operation speed of the who ...

All the trees we use in this article are user JSON, as shown in the figure below {"username": "zhangsan","age": 20} {"username": "lisi","age": 21} {"username": "wangwu","age": 19} Custom UDF Introduction to UDF UDF: enter a line and return a result For one-to-one relationship, if you put a value into a function, you will return ...

catalogue Requirements: query a given number of tables, several of which are used in the code Transformation concept: given key Word, several hits in the log file Type selection: Spark parsing keyword list rdd1;Spark parses file directory data rdd2; rdd1 join rdd2(broadcast) Upper Code: Step 1: create SparkContext Step 2: read the file ...

catalogue 1, Entry class - SparkSubmit 2, SparkApplication startup - JavaMainApplication, YarnClusterApplication 3, SparkContext initialization 4, YarnClientSchedulerBackend and YarnClusterSchedulerBackend initialization 5, ApplicationMaster startup 6, Spark on Yan task submission process summary 1, Entry class - SparkSubmit When sub ...