Programmer Think

catalogue 1, Entry class - SparkSubmit 2, SparkApplication startup - JavaMainApplication, YarnClusterApplication 3, SparkContext initialization 4, YarnClientSchedulerBackend and YarnClusterSchedulerBackend initialization 5, ApplicationMaster startup 6, Spark on Yan task submission process summary 1, Entry class - SparkSubmit When sub ...

I RDD overview 1.1 what is RDD RDD (Resilient Distributed Dataset) is called elastic distributed dataset. It is the most basic data abstraction in Spark. Code is an abstract class, which represents an elastic, immutable, partitioned collection in which the elements can be calculated in parallel. 1.2 RDD features (1) Flexibility Elastici ...

preface The previous project was based on springboot and integrated spark, running on standalone. I once wrote a blog, link: https://blog.csdn.net/qq_41587243/article/details/112918052?spm=1001.2014.3001.5501 The same scheme is used now, but spark is submitted on the production environment yarn cluster, and kerbores verification is required, ...

1, The origin of the problem In the query of elastic search, we usually set the index to search directly through the URL; If we need to query more indexes and have no rules, we will face an embarrassing situation, exceeding the length limit of the URL; 2, Test environment elasticsearch 6.8.12 test data Add three test indexes, one document ...

Apache Hadoop ##Background With the development requirements of information internet and Internet of things, the trend of interconnection of all things is imperative. This leads to the evolution of architecture from a single architecture to a highly concurrent distributed architecture. Data storage also began to evolve from the original stand ...

3, RDD action operator The so-called action operator is actually a method that can trigger Job execution 1,reduce  function signature def reduce(f: (T, T) => T): T  function description Aggregate all elements in RDD, first aggregate data in partitions, and then aggregate data between partitions val rdd: RDD[Int] = sc.m ...

preface This article belongs to the column "100 problems to solve the installation and deployment of big data". This column is original by the author. Please indicate the source of quotation. Please help point out the deficiencies and errors in the comment area. Thank you! For the directory structure and references of this column ...

background The project needs to deal with many files, and some files have a large number of GB. Therefore, considering that such files are specially written for Spark program processing, for the unified processing of programs, it is necessary to call Spark jobs in code to handle large files. Implementation scheme After investigation, it is foun ...

1. Differences between Spark on Hive and Hive on Spark 1)Spark on Hive Spark on Hive is Hive's only storage role and Spark is responsible for sql parsing optimization and execution. You can understand that Spark uses Hive statements to manipulate Hive tables through Spark SQL, and Spark RDD runs at the bottom. The steps are as follows: ...

Variables and data types notes As like as two peas in Java, there is not much to say here. // Single-Line Comments /* multiline comment */ /* * Documentation Comments */ variable and constant Constant: a variable that will not be changed during program execution // variable // var variable name [: variable type] = initial value var i ...