Programmer Think

spark Learning Notes - core operator (2) distinct operator /** * Return a new RDD containing the distinct elements in this RDD. */ def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope { def removeDuplicatesInPartition(partition: Iterator[T]): Iterator[T] = { // Create an instance of extern ...

This article is about Learning Guide for Big Data Specialists from Zero (Full Upgrade) Added in part by Haop. 1. Template Virtual Machine Environment Preparation 0) Install template virtual machine, IP address 192.168.10.100, host name hadoop100, memory 4G, hard disk 50G 1) The configuration requirements for hadoop100 virtual machine are a ...

1. Internal class 1.1 basic use of internal classes (understanding) Inner class concept Define a class in a class. For example, if a class B is defined inside a Class A, class B is called an inner class Internal class definition format Format & ex amp le: /* Format: class External class name{ Modifier class intern ...

Summary of practical applications of Kafka, RabbitMQ and RockedMQ 1.Kafka Combined with the use cases on the official website, this paper records the examples and practical application of the three mainstream mq.This article does not cover the installation and configuration of relevant environments, but involves more comprehensive codes (inclu ...

1.Four cornerstones of Flink Flink can be so popular without its four most important cornerstones: Checkpoint, State, Time, Window. ◼ Checkpoint This is one of Flink's most important features. Flink implements a distributed and consistent snapshot based on Handy-Lamport algorithm, which provides consistent semantics. The Chandy-Lamport a ...

Next, we will introduce some more detailed implementation details. Shuffle is undoubtedly a key point of performance tuning. This paper will deeply analyze the implementation details of Spark Shuffle from the perspective of source code implementation. The upper boundary of each Stage requires either reading data from external storage or r ...

1, Foreword Two days ago, we did an import function. The import started very slowly. It took more than one minute to import 2w pieces of data. Later, we optimized it bit by bit, from directly linking the list into Mysql, assigning the list into Mysql, and multi threading the list into Mysql. Time is getting less and less. It was very cool, and ...

When the requirements of an application are relatively simple, there may not be many operators involved in data conversion, but when the requirements of the application become more and more complex, the number of operators in a Job may reach dozens or even hundreds. With so many operators, the whole application will become very complex, So it w ...

The next day, I started learning Python from scratch. In fact, I made a rapid progress yesterday. I didn't stop until 3-7. Because I wanted to use the while loop, I didn't know much about Python's while (in fact, I didn't want to learn any more because it was too late). Then I finished 3-7 this morning. 3-1 Name: store the names of some friend ...

Baidu sexy beauty wallpaper to understand, for nothing else, is to be able to better learn Python! Do you believe that? As long as I see the website with photos of my sister, I just want to download them in bulk!   Why to learn about web crawlers(1) Learning the web crawler, you can customize a search engine to better understand the principle o ...