Programmer Think

I RDD overview 1.1 what is RDD RDD (Resilient Distributed Dataset) is called elastic distributed dataset. It is the most basic data abstraction in Spark. Code is an abstract class, which represents an elastic, immutable, partitioned collection in which the elements can be calculated in parallel. 1.2 RDD features (1) Flexibility Elastici ...

preface The previous project was based on springboot and integrated spark, running on standalone. I once wrote a blog, link: https://blog.csdn.net/qq_41587243/article/details/112918052?spm=1001.2014.3001.5501 The same scheme is used now, but spark is submitted on the production environment yarn cluster, and kerbores verification is required, ...

3, RDD action operator The so-called action operator is actually a method that can trigger Job execution 1,reduce  function signature def reduce(f: (T, T) => T): T  function description Aggregate all elements in RDD, first aggregate data in partitions, and then aggregate data between partitions val rdd: RDD[Int] = sc.m ...

background The project needs to deal with many files, and some files have a large number of GB. Therefore, considering that such files are specially written for Spark program processing, for the unified processing of programs, it is necessary to call Spark jobs in code to handle large files. Implementation scheme After investigation, it is foun ...

1. Differences between Spark on Hive and Hive on Spark 1)Spark on Hive Spark on Hive is Hive's only storage role and Spark is responsible for sql parsing optimization and execution. You can understand that Spark uses Hive statements to manipulate Hive tables through Spark SQL, and Spark RDD runs at the bottom. The steps are as follows: ...

By default, Spark can divide a job into multiple tasks and send it to the Executor node for parallel computing. The number of tasks that can be calculated in parallel is called parallelism. This number can be specified when building the RDD. However, the number of splitting tasks is not necessarily equal to the number of tasks executed in paral ...

Reference link https://www.bilibili.com/video/BV11A411L7CK?p=11 Spark operating environment As a data processing framework and computing engine, Spark is designed to run in all common cluster environments. The mainstream environment in domestic work is yard, but the container environment is becoming more and more popular Local mode The s ...

1. Operation mechanism of Spark On YARN deployment mode After the task is submitted, start the Driver;Driver registers the application with the cluster manager;The cluster manager allocates the Executor according to the configuration file of this task and starts it;The Driver starts to execute the main function. When the action operator is ...

Error log: 1. Error log: ​ WARN TaskSetManager: Lost task 29.0 in stage 22.0 (TID 1851, wn108-cdlcns.bjduloineequ3adfbkrpgi4p2c.shax.internal.chinacloudapp.cn, executor 25): FetchFailed(BlockManagerId(2, wn107-cdlcns.bjduloineequ3adfbkrpgi4p2c.shax.internal.chinacloudapp.cn, 7447, None), shuffleId=12, mapId=4, reduceId=29, message= org.apach ...

Local operation mode Basic introduction Spark's Local operation mode is also called Local operation mode and pseudo distributed mode. This is called Local mode because all spark processes in this mode run in the virtual machine of a Local machine without any resource manager. It mainly uses multiple threads of a single machine to simulate spa ...