Programmer Think

1: RDD operator RDD operators, also known as RDD methods, are mainly divided into two categories. Conversion operator and action operator. 2: RDD conversion operator According to different data processing methods, the operators are divided into value type, double value type and key value type 2.1: map value conversion package com.atg ...

Unit 8: time series analysis in Google Earth engine 1 Introduction In this module, we will discuss the following concepts: Process remote sensing images of the ocean. Create a video from an image time series. Time series analysis in GEE. Add basic elements to the graphical user interface. 2 background The Deepwat ...

Unit 1: introduction to Google Earth engine Developed by the Colorado State University graduate degree program in ecology and the laboratory of natural resource ecology Authors: Peder Engelstad, Dan Carver Updated may 2021 1 Introduction2 background3 start using Google Earth engine 3.1 browsing user interface3.2 introducti ...

preparation # Switch the directory to / export / and empty the contents cd /export/ && rm -rf /export/* && tree # Add a directory and switch directories and add files mkdir -p /export/aaa/bbb/ccc/ddd/eee/ touch /export/aaa/bbb/ccc/ddd/eee/abc.txt touch /export/aaa/123.txt touch /export/aaa/312.txt # View the contents of ...

Operator of RDD What is the 1-operator? API, method, behavior What are the classes of 2-operators- transformation and action 3-transformation features: convert to new RDD and delay loading What operators does - transformation have- See the table, such as map filter, etc - transformation continue classification eg: glom - Elements of each partit ...

Big data, Spark, Hadoop, python Big data Hadoop installation and configuration 1, Hadoop pseudo distributed configuration 1. Create Hadoop user: sudo useradd -m hadoop -s /bin/bash # Create hadoop user sudo passwd hadoop # Change Password sudo adduser hadoop sudo # Add administrator privileges Log out and log in ...

brief introduction Last article Playing Hudi Docker Demo based on Ubuntu (2) -- writing test data to Kafka Describes how to write test data to fkaka cluster. This article describes how to use Spark to consume Kafka data and write the data to HDFS. Hudi is introduced into Spark in the form of Jar package. Types of Hudi tables and queries Tabl ...

Record Basic environment JDK8Python3.7 Setting up Spark environment in Window First install JDK8 and python 3, which will not be repeated here Install Hadoop 2 seven Download address: http://archive.apache.org/dist/hadoop/core/hadoop-2.7.7/hadoop-2.7.7.tar.gz decompression Download winutils of hadoop: https://github.com/stevelou ...

edition Version upgrade instructions: prevent risks caused by log4j vulnerabilities Unified Version Description: unify the version to prevent unnecessary accidents Version selection Description: elasticsearch: 7.16.2 logstash: 7.16.2 file beat: 7.16.2 Download instructions Enter the official website es official website Select the required ...

catalogue preparation in advance Create user Download the installation package of ElasticSeach. This case uses version 6.2.4 Modify the configuration item JVM options Modify the configuration item elasticsearch yml Resolve error reporting Start Elasticsearch 1.3.1. Error 1: kernel too low 1.3.2. Error 2: insufficient file permissions ...