Complete use of Scala collection of big data (Chapter 7)

Posted by dhcrusoe on Fri, 25 Feb 2022 05:20:39 +0100

1, Collection introduction

1) Scala's collections fall into three categories: sequence Seq, Set and Map. All collections are extended from Iterable
Characteristics.

2) For almost all collection classes, Scala provides both variable and immutable versions, which are located in the following two places:
Package

Immutable set: Scala collection. immutable
Variable set: Scala collection. mutable

3) Scala immutable set means that the set object cannot be modified. Each modification will return a new object, and
The original object is not modified. Similar to String objects in java

4) A variable set is a set that can directly modify the original object without returning a new object. similar
StringBuilder object in java

**Suggestion: when operating a set, use symbols for immutability and methods for mutability**

1. Immutable set inheritance graph

1) Set and Map are also collections in Java
2) Seq is not available in Java. We found that the List belongs to SEQ, so the List here is not the same as Java
Concept
3) Our previous for loop has a 1 to 3, which is the Range under IndexedSeq
4) String also belongs to IndexedSeq
5) We found that classical data structures, such as Queue and Stack, belong to linearseq (linear sequence). 6) note that the Map system in scala has a SortedMap, which shows that Scala's Map can support sorting
7) The difference between IndexedSeq and LinearSeq:

(1) IndexedSeq searches and locates by index, so it is fast. For example, String is an index set, which can be located by index
(2) LinearSeq is linear, that is, it has the concept of head and tail. This data structure is generally searched through history

2. Variable set inheritance graph

2, Array

1. Immutable array

1) The first way is to define an array

Definition: val arrar1 = new array Int
(1) new is the keyword
(2) [Int] specifies the data type that can be stored. If you want to store Any data type, specify Any
(3) (10) indicates the size of the array, which cannot be changed after confirmation

2) Case practice

package chapter07

/**
 * Create array
 * Immutable arrays are recommended in Scala
 */
object Test01_ImmutableArray {
  def main(args: Array[String]): Unit = {
    //1. Create array
    val arr: Array[Int] = new Array[Int](5)

    //Another way to create
    val arr2 = Array(12, 32, 23, 13, 16)

    //2. Access and modify elements
    println(arr(0))
    println(arr(1))
    println(arr(4))

    println("++++++++++++++++++++++++++++++++++")
    arr(0) = 12
    arr(4) = 57

    println(arr(0))
    println(arr(1))
    println(arr(4))


    //3. Traversal of array
    //3.1 ordinary for loop - > until does not contain
    for (i <- 0 until arr.length) {
      println(arr(i))
    }
    println("+++++++++++++++ordinary for loop+++++++++++++++++++")
    for (i <- arr.indices) {
      println(arr(i))
    }

    //3.2 directly facilitate all elements and enhance for loop
    println("+++++++++++++++enhance for loop+++++++++++++++++++")
    for (elem <- arr2) {
      println(elem)
    }

    // 3) Iterator
    println("+++++++++++++++iterator +++++++++++++++++++")
    val iterator = arr2.iterator
    while (iterator.hasNext)
      println(iterator.next())

    println("+++++++++++++++call foreach+++++++++++++++++++")
    //4. Call foreach method
    arr2.foreach((elem: Int) => {
      println(elem)
    })

    println("+++++++++++++++call foreach2+++++++++++++++++++")
    arr2.foreach(println)

    //Splice string
    println(arr2.mkString("--"))

    println("--------------Add element------------------------")
    //5. Add element
    val newArray1 = arr2.:+(73) // .: + means appending the end
    println(arr2.mkString("--")) //Because it is immutable, it will not be added

    println(newArray1.mkString("--")) //You can return a new array display

    val newArray2 = newArray1.+:(30) //. +: represents the addition of the front
    println(newArray2.mkString("--"))


    val newArray3 = newArray2 :+ 30 //. +: represents the addition of the front

    /**
     * TODO
     * 1,.:+ Represents the end of the append
     * 2,.+: Represents the addition of the front
     */
    val newArray4 = 19 +: 29 +: newArray3 :+ 26 :+ 73
    println(newArray4.mkString("--"))
  }
}

3) The second way is to define an array

val arr1 = Array(1, 2)
(1) When defining an array, initial values are assigned directly
(2) Use the apply method to create an array object

4) Case practice

object TestArray{
 def main(args: Array[String]): Unit = {
 var arr02 = Array(1, 3, "bobo")
 println(arr02.length)
 for (i <- arr02) {
 println(i)
 }
 } }

2. Variable array

1) Defining variable length arrays

val arr01 = ArrayBuffer[Any](3, 2, 5)
(1) [Any] store Any data type
(2) (3, 2, 5) three initialized elements
(3) Arraybuffer needs to introduce Scala collection. mutable. ArrayBuffer

2) Case practice

(1) ArrayBuffer is an ordered collection
(2) The add element uses the append method (), which supports variable parameters

package chapter07

import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer

/**
 * Array, variable array
 *
 */
object Test02_ArrayBuffer {
  def main(args: Array[String]): Unit = {
    //TODO 1. Create variable array
    println("--------TODO 1,Create variable array-----------------")
    val arr1: ArrayBuffer[Int] = new ArrayBuffer[Int]()
    val arr2 = ArrayBuffer(23, 44, 66, 77, 31)

    println(arr1)
    println(arr1.mkString(","))
    println(arr2)

    println("--------TODO 2,Access element-----------------")
    //TODO 2. Access element
    println(arr2(1))
    arr2(1) = 39
    println(arr2(1))

    //    arr1(1) = 1
    //    println(arr1(1))

    println("-------TODO 3,Add element-------------------------")

    val newArr1 = arr1 :+ 15
    println(arr1)
    println(newArr1)
    println(newArr1 == arr1)

    // +=Equivalent to adding
    val newArr2 = arr1 += 19
    println(arr1)
    println(newArr2)
    println(newArr2 == arr1)
    newArr2 += 13
    println(arr1)

    //Append +:
    77 +=: arr1
    println(arr1)
    println(newArr2)

    arr1.append(36)
    arr1.prepend(11, 34)
    arr1.insert(1, 35, 59)
    println(arr1)

    arr1.insertAll(2, newArr1)
    arr1.prependAll(newArr2)

    println(arr1)
    println("-------TODO 4,Delete element-------------------")
    arr1.remove(3)
    println(arr1)

    arr1.remove(0, 10)
    println(arr1)

    //Find, delete, and do nothing if you can't find it
    arr1 -= 13
    println(arr1)

    println("-------TODO 5,Variable array to immutable array-------------------")
    val arr: ArrayBuffer[Int] = ArrayBuffer(23, 56, 46)
    val newArray: Array[Int] = arr.toArray
    println(newArray.mkString(","))
    println(arr)

    println("-------------Convert an immutable array to a mutable array---------------")
    val buffer: mutable.Buffer[Int] = newArray.toBuffer
    println(buffer)
    println(newArray)
  }

}

3. Conversion between immutable array and variable array

1) Explain

arr1.toBuffer / / convert immutable array to variable array
arr2.toArray / / convert variable array to immutable array
(1)arr2. The returned result of toArray is an immutable array, and arrar2 itself does not change
(2)arr1. The returned result of tobuffer is a variable array, and Arr1 itself does not change

2) Case practice

package chapter07

import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer

/**
 * Array, variable array
 *
 */
object Test02_ArrayBuffer {
  def main(args: Array[String]): Unit = {
    //TODO 1. Create variable array
    println("--------TODO 1,Create variable array-----------------")
    val arr1: ArrayBuffer[Int] = new ArrayBuffer[Int]()
    val arr2 = ArrayBuffer(23, 44, 66, 77, 31)

    println(arr1)
    println(arr1.mkString(","))
    println(arr2)

    println("--------TODO 2,Access element-----------------")
    //TODO 2. Access element
    println(arr2(1))
    arr2(1) = 39
    println(arr2(1))

    //    arr1(1) = 1
    //    println(arr1(1))

    println("-------TODO 3,Add element-------------------------")

    val newArr1 = arr1 :+ 15
    println(arr1)
    println(newArr1)
    println(newArr1 == arr1)

    // +=Equivalent to adding
    val newArr2 = arr1 += 19
    println(arr1)
    println(newArr2)
    println(newArr2 == arr1)
    newArr2 += 13
    println(arr1)

    //Append +:
    77 +=: arr1
    println(arr1)
    println(newArr2)

    arr1.append(36)
    arr1.prepend(11, 34)
    arr1.insert(1, 35, 59)
    println(arr1)

    arr1.insertAll(2, newArr1)
    arr1.prependAll(newArr2)

    println(arr1)
    println("-------TODO 4,Delete element-------------------")
    arr1.remove(3)
    println(arr1)

    arr1.remove(0, 10)
    println(arr1)

    //Can't find or delete anything
    arr1 -= 13
    println(arr1)

    println("-------TODO 5,Variable array to immutable array-------------------")
    val arr: ArrayBuffer[Int] = ArrayBuffer(23, 56, 46)
    val newArray: Array[Int] = arr.toArray
    println(newArray.mkString(","))
    println(arr)

    println("-------------Convert an immutable array to a mutable array---------------")
    val buffer: mutable.Buffer[Int] = newArray.toBuffer
    println(buffer)
    println(newArray)
  }

}

4. Multidimensional array

1) Multidimensional array definition

val arr = Array.ofDimDouble
Description: there are three one-dimensional arrays in the two-dimensional array, and each one-dimensional array has four elements

2) Case practice

package chapter07

/**
 * Multidimensional array
 */
object Test03_MulArray {
  def main(args: Array[String]): Unit = {
    //1. Create multidimensional array
    val array: Array[Array[Int]] = Array.ofDim[Int](2, 3)

    //2. Access element
    array(0)(2) = 19
    array(1)(0) = 25

    println(array.mkString(","))

    println("-------Abbreviation 1----------------")
    for (i <- 0 until array.length; j <- 0 until array(i).length) {
      println(array(i)(j))
    }


    println("-------Cycle two----------------")

    for (i <- array.indices; j <- array(i).indices) {
      println(array(i)(j) + "\t")
      if (j == array(i).length - 1) println()
    }

    println("-------Abbreviation----------------")
    array.foreach(line => line.foreach(println))

    //Abbreviation_ replace
    println("-------_Replace abbreviation----------------")
    array.foreach(_.foreach(println))

  }
}

3, List list

1. Immutable List

1) Explain

(1)List The default is immutable set
(2)Create a List((data in order and repeatable)
(3)ergodic List
(4)List Add data
(5)Merging between sets: splitting a whole into individuals is called flattening
(6)Get the specified data
(7)Empty set Nil

2) Case practice

package chapter07

/**
 * 1)explain
 * (1)List The default is immutable set
 * (2)Create a List (the data is sequential and repeatable)
 * (3)Traverse List
 * (4)List Add data
 * (5)Merging between sets: splitting a whole into individuals is called flattening
 * (6)Get the specified data
 * (7)Empty set Nil
 */
object Test04_List {
  def main(args: Array[String]): Unit = {
    //1. Create
    var list = List(23, 45, 78)
    println(list)

    //2. Index assignment failed
    //list(1) = 20

    //3. Access the loop of traversal and foreach
    list.foreach(e => {
      println(e)
    })

    //4. Add element, before + before corresponding, after + after corresponding
    val list1 = 10 +: list
    val list2 = list :+ (24)

    println(list1)
    println(list2)

    println("----------5, .:: Add forward-----------------")
    val list4 = list2.::(51)
    println(list4)

    println("----------6, Nil.:: This is equivalent to creating a new array-----------------")
    val list5 = Nil.::(51)
    println(list5)

    println("----------7,Extension adding elements is another way to create a new list -----------------")
    val list6 = 32 :: Nil
    val list7 = 17 :: 28 :: 59 :: 16 :: Nil

    println(list6)
    println(list7)

    println("----------8,Merge list -----------------")

    val list8 = list6 :: list7
    println(list8)   //List(List(32), 17, 28, 59, 16)

    val list9 = list6 ::: list7
    println(list9) //List(32, 17, 28, 59, 16)

    val list10 = list6 ++ list7
    println(list10) //List(32, 17, 28, 59, 16)
  }
}

2. Variable ListBuffer

1) Explain

(1) Create a variable collection ListBuffer
(2) Adding data to a collection
(3) Print set data

2) Case practice

package chapter07

import scala.collection.mutable.ListBuffer

/**
 * Scala aggregate
 * list
 * Variable list
 */
object Test05_ListBuffer {
  def main(args: Array[String]): Unit = {
    //1. Create variable list
    val list1: ListBuffer[Int] = new ListBuffer[Int]();
    val list2 = ListBuffer(12, 45, 23, 12)

    println(list1)
    println(list2)

    println("-------------Add element------------------")
    //2. Add element
    list1.append(13, 56)
    list2.prepend(20)

    //Specify index addition
    list1.insert(1, 19, 22)

    println(list1)
    println(list2)

    println("----------Add one in front-----Add two after----------------")
    31 +=: 96 +=: list1 += 25 += 11
    println(list1)

    println("-------3,Merge elements---------------")
    //3. Merge elements
    val list3 = list1 ++ list2
    println(list3)

    // ↓ abbreviation
    println("-------Abbreviation---------------")
    list1 ++= list2
    println(list1)
    println(list2)

    println("-------4,Modify element---------------")
    //4. Modify element
    list2(3) = 30
    list2.update(0, 90)
    println(list2)

    println("-------5,Delete element---------------")

    list2.remove(2)
    list2 -= 23
    println(list2)

  }

}

4, Set set

By default, Scala Use immutable sets. If you want to use variable sets, you need to reference them
scala.collection.mutable.Set package

1. Immutable Set

1) Explain

(1) Set is an immutable set by default, and the data is out of order
(2) Data cannot be duplicated
(3) Traversal set

2) Case practice

package chapter07

/**
 * Set aggregate
 * Immutable set
 *
 * If the contents of the original set are not changed, the result is a new set object
 *
 * Make changes that are variable and directly based on the original
 */
object Test06_ImmutableSet {
  def main(args: Array[String]): Unit = {

    //1. Create duplicate auto remove
    val set1 = Set(12, 23, 34, 56, 12, 23, 12)
    println(set1)


    //2. Add element
    println("---------2,Add element------------")
    val set2 = set1 + 129
    println(set1)
    println(set2)


    //3. Merge elements
    println("---------3,Merge elements------------")
    val set3 = Set(19, 13, 23, 53, 67, 99)

    val set4 = set2 ++ set3
    println(set2)
    println(set3)
    println(set4)


    //4. Delete element
    println("---------4,Delete element------------")
    val set5 = set3 - 13
    println(set3)
    println(set5)
  }
}

2. Variable mutable Set

1) Explain

(1) Create mutable set Set
(2) Print set
(3) Add element to collection
(4) Add an element to the collection and return a new Set
(5) Delete data

2) Case practice

package chapter07

import scala.collection.mutable

/**
 * Set aggregate
 * Variable Set
 *
 * Immutable collection type recommendation: symbolic method
 * English names are recommended for variable sets
 */
object Test07_MutableSet {
  def main(args: Array[String]): Unit = {
    val set1: mutable.Set[Int] = mutable.Set(12, 23, 53, 11, 13, 12, 78)
    println(set1)

    println("=======================")

    //2. Add element
    val set2 = set1 + 11
    println(set1)
    println(set2)

    set1 += 11
    println(set1)

    //You can judge whether the element is added successfully
    val bool = set1.add(10)
    println(bool)
    println(set1)

    //Judge whether it already exists or not added
    val bool1 = set1.add(10)
    println(bool1)
    println(set1)

    println("=======Delete element================")

    set1 -= 11
    println(set1)

    val boo2 = set1.remove(10)
    println(boo2)
    println(set1)

    val bool3 = set1.add(10)
    println(bool3)
    println(set1)

    println("=======Merge and connect Set================")

    val set3 = mutable.Set(13, 12, 13, 27, 98, 29)

    val set4= set1 ++ set3

    //++=Who calls this symbol and who changes it

    set1 ++= set3
    println(set1)
    println(set3)
    println(set4)

  }
}

5, Map collection

Similar to Java, Map in Scala is also a hash table, which stores key value pairs
mapping

1. Immutable Map

1) Explain

(1) Create immutable set Map
(2) Cyclic printing
(3) Access data
(4) If the key does not exist, 0 is returned

2) Case practice

package chapter07

/**
 * Scala aggregate
 * Map Set immutable
 */
object Test08_ImmutableMap {
  def main(args: Array[String]): Unit = {

    //1. Create Map
    val map1: Map[String, Int] = Map("a" -> 13, "b" -> 24, "hello" -> 80)
    println(map1)
    println(map1.getClass)

    println("------2,Traversal element----------------")
    //2. Traversal element
    map1.foreach(println)
    map1.foreach((kv: (String, Int)) => println(kv))

    //3. Get all keys or value s of the set in the Map
    println("-----------3,take Map All sets in Key perhaps value------------")
    for (key <- map1.keys) {
      println(s"$key --->${map1.get(key)}")
    }

    //4. Accessing the value of a key
    println("a :" + map1.get("a").get)
    println("c :" + map1.get("c"))
    println("c :" + map1.getOrElse("c", 0))

  }
}

2. Variable Map

1) Explain

(1) Create variable set
(2) Print set
(3) Add data to collection
(4) Delete data
(5) Modify data

2) Case practice

package chapter07

import scala.collection.mutable

/**
 * Map aggregate
 * Variable Map
 */
object Test09_MutableMap {
  def main(args: Array[String]): Unit = {
    //1. Create Map
    val map1: mutable.Map[String, Int] = mutable.Map("a" -> 13, "b" -> 53, "hello" -> 23)
    println(map1)
    println(map1.getClass)

    println("=============================")
    //2. Add element
    map1.put("c", 5)
    map1.put("d", 6)
    println(map1)

    map1 += (("e", 7))
    println(map1)

    println("=============================")

    //3. Delete element
    println(map1("c"))
    map1.remove("c")
    println(map1)
    println(map1.getOrElse("c", 0))


    //4. Modify element
    println("===========4,Modify element==================")
    map1.update("c", 5)
    map1.update("e", 10)
    println(map1)

    map1 += (("e", 7))
    println(map1)


    println("=============================")
    val map2: Map[String, Int] = Map("aaa" -> 11, "b" -> 29, "hello" -> 5)
    map1 ++= map2
    println(map1)
    println(map2)

    println("-------------------------------")
    val map3: Map[String, Int] = map2 ++ map1
    println(map1)
    println(map2)
    println(map3)
  }
}

6, Tuple

1) Explain

Tuple can also be understood as a container, which can store all kinds of the same or different types of data. To put it simply, it is to encapsulate multiple unrelated data into a whole, which is called tuple.
Note: a tuple can have a maximum of 22 elements.

2) Case practice

(1) Way to declare tuples: (element 1, element 2, element 3)
(2) Access tuple
(3) The key value pairs in Map are actually tuples, but the number of elements of tuples is 2, which is called duality

package chapter07

/**
 * Scala aggregate
 * tuple
 *
 */
object Test10_Tuple {
  def main(args: Array[String]): Unit = {
    //1. Create tuple
    val tuple: (String, Int, Char, Boolean) = ("hello", 100, 'a', true)

    println(tuple)

    //2. Access data
    println(tuple._1)
    println(tuple._2)
    println(tuple._3)
    println(tuple._4)

    //3. Traversing tuple data
    println("3,Traversing tuple data")
    for (elem <- tuple.productIterator) {
      println(elem)
    }

    //4. Nested tuple
    println("---------Nested tuple-------------")
    val muTuple = (12, 0.3, "hello", (23, "scala"), 29)
    println(muTuple._4._2)
  }
}

7, Set common functions

1. Basic properties and common operations

1) Explain

(1)Get collection length
(2)Get collection size
(3)Loop traversal
(4)iterator 
(5)Generate string
(6)Include

2) Case practice

package chapter07


/**
 *
 *  General properties and methods
 * Linear sequences can obtain the length
 */
object Test11_CommonOp {
  def main(args: Array[String]): Unit = {
    var list = List(1, 3, 5, 7, 89)

    var set = Set(23, 13, 45, 68)

    //1. Get collection length
    println(list.length)
    //2. Get collection size
    println(set.size)

    //3. Loop traversal
    println("list...")
    for (elem <- list) {
      println(elem)
    }


    println("set...")
    set.foreach(println)

    //4. Iterator

    //5. Generate string
    println(list)
    println(set)
    println(list.mkString("----"))

    //6. Include

    println(list.contains(23))
    println(set.contains(23 ))
  }
}

2. Derived set

1) Explain

(1) Gets the header of the collection
(2) Get the tail of the collection (either the head or the tail)
(3) Set last data
(4) Set initial data (excluding the last one)
(5) Reverse
(6) Take the first (last) n elements
(7) Remove the first (last) n elements
(8) Union
(9) Intersection
(10) Difference set
(11) Zipper
(12) Sliding window

2) Case practice

package chapter07

/**
 * Derived set
 */
object Test12_DerivedCollection {
  def main(args: Array[String]): Unit = {

    val list1 = List(1, 3, 5, 7, 289)
    val list2 = List(3, 7, 2, 45, 4, 8, 19)
    //(1) Gets the header of the collection

    println(list1.head)

    //(2) Get the tail of the collection (either the head or the tail)
    println(list1.tail)

    //(3) Set last data
    println(list2.last)

    //(4) Set initial data (excluding the last one)
    println(list2.init)
    //(5) Reverse
    println(list2.reverse)

    //(6) Take the first (last) n elements
    println(list1.take(3))
    println(list1.takeRight(4))

    //(7) Remove the first (last) n elements
    println("Remove front (rear) n Elements")
    println(list1.drop(3))
    println(list1.takeRight(4))

    //(8) Union
    val union = list1.union(list2)
    println("union: " + union)

    println(list1 ::: list2)

    //If set is a union, it will be de duplicated
    println("If it is set If you do Union, you will go to duplicate")
    val set1 = Set(1, 3, 5, 7, 289)
    val set2 = Set(3, 7, 2, 45, 4, 8, 19)
    println(set1.union(set2))
    println(set1 ++ set2)

    //(9) Intersection
    println("intersection")
    val intersect = list1.intersect(list2)
    println("intersect: " + intersect)

    //(10) Difference set
    val diff1 = list1.diff(list2)
    val diff2 = list2.diff(list1)
    println("diff1 " + diff1)
    println("diff2 " + diff2)

    //(11) Zipper
    println("zipper")
    println("zip: " + list1.zip(list2))
    println("zip: " + list2.zip(list1))

    //(12) Sliding window
    println("sliding window")
    for (elem <- list1.sliding(3)) {
      println(elem)
    }
    
    println("Sliding windows slide once every two")
    for (elem <- list1.sliding(3,2)) {
      println(elem)
    }

  }
}

3. Set calculation simple function

1) Explain

(1) Sum
(2) Product
(3) Maximum
(4) Minimum value
(5) Sort

2) Practical operation

package chapter07

/**
 * Set common functions
 * Simple calculation function
 */
object Test13_SimpleFunction {
  def main(args: Array[String]): Unit = {
    val list = List(1, 2, 4, 8, 5, -3)

    val list2 = List(("a", 5), ("b", 1), ("c", 8), ("d", 2), ("e", -3), ("f", 4))

    //1. Sum
    var sum = 0
    for (elem <- list) {
      sum += elem
    }
    println(sum)
    println(list.sum)

    //2. Product

    println(list.product)

    //3. Maximum
    println(list.max)

    println(list2.maxBy((tuple: (String, Int)) => tuple._2))
    //Take the second element
    println(list2.maxBy(_._2))

    //4. Minimum value
    println(list.min)
    println(list2.minBy(_._2))

    //5. Sort
    println("sort")
    // 5.1 sorted
    val sortedList = list.sorted
    println(sortedList)

    println("Sort in reverse order from large to small")
    println(list.sorted.reverse)

    //Pass in implicit parameters
    println(list.sorted(Ordering[Int].reverse))
    println(list2.sorted)

    //5.2 sortBy  //Ordering[Int].reverse
    println(list2.sortBy(_._2))
    println(list2.sortBy(_._2)(Ordering[Int].reverse))

    //5.3 sortWith / / sort from small to large
    println(list.sortWith((a: Int, b: Int) => {
      a < b
    }))

    //Less than sort
    println(list.sortWith(_ < _))

    //Greater than sort
    println(list.sortWith(_ > _))


  }
}

(1)sorted
Sort a collection naturally by passing implicit Ordering
(2)sortBy
Sort one or more attributes by their type.
(3)sortWith
Based on function sorting, a comparator function is used to realize the logic of user-defined sorting.

4. Set computing advanced functions

1) Explain

(1) Filter
Traverse a collection and get the elements that meet the specified conditions from it to form a new collection

(2) Transform / map
Map each element in the collection to a function

(3) Flattening

(4) Flattening + mapping note: flatMap is equivalent to map operation first and then flatten operation
The child element of each element in the collection maps to a function and returns a new collection

(5) Group
Groups the elements of the collection according to the specified rules

(6) Simplification (reduction)

(7) Fold

2) Practical operation

package chapter07

/**
 * Set computing advanced functions
 */
object Test14_HighLevelFunction_Map {
  def main(args: Array[String]): Unit = {
    var list = List(1, 2, 3, 4, 5, 6, 7, 8, 9)

    //1. Filter
    //Select even number
    val evenList = list.filter((elem: Int) => {
      elem % 2 == 0
    })
    println(evenList)

    //Select odd number
    println(list.filter(_ % 2 == 1))


    //2,map
    //Multiply each number in the set by two
    println("---------------------------")
    println(list.map(_ * 2))
    println(list.map(x => x * x))


    //3. Flattening
    println("delayering")
    val nestedList: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 9))
    val ints = nestedList(0) ::: nestedList(1) ::: nestedList(2)
    println(ints)

    val flatten = nestedList.flatten
    println(flatten)
    println("============================")


    //4. Flat mapping
    println("Flat mapping")
    //Segment a set of strings and save them as a list of words
    val strings: List[String] = List("hello word", "hello scala", "hello java", "we study")
    val stringses: List[Array[String]] = strings.map(_.split(" ")) //participle
    val flattenList = stringses.flatten
    println(flattenList)


    val strings1 = strings.flatMap(_.split(" "))
    println(strings1)

    //5. Group by
    //Divided into odd and even groups
    val groupMap: Map[Int, List[Int]] = list.groupBy(_ % 2)
    val groupMap2: Map[String, List[Int]] = list.groupBy(data => {
      if (data % 2 == 0) "even numbers" else "Odd number"
    })

    println(groupMap)
    println(groupMap2)


    //6. Given a set of words, group them according to the first letter of the word
    val strings2 = List("china", "america", "alice", "canada", "cary", "bob", "japan")
    println(strings2.groupBy(_.charAt(0)))
  }
}

3) Reduce method and Fold method

Reduce: aggregate the data in the collection through the specified logic, so as to reduce the data and minimize the cost
Finally get the results.
Case practice

package chapter07

/**
 * Advanced computing function
 * Set conversion operation Reduce
 */
object Test15_HighLevelFunction_Reduce {
  def main(args: Array[String]): Unit = {
    val list = List(1, 2, 3, 4)

    //1,Reduce
    println(list.reduce(_ + _))
    println(list.reduceLeft(_ + _))
    println(list.reduceRight(_ + _))

    println("------------------------------------")


    //reduce can think that the initial value is the first number
    val list2 = List(3, 4, 5, 8, 10)
    println(list2.reduce(_ - _)) //-24
    println(list2.reduceLeft(_ - _))
    println(list2.reduceRight(_ - _))

    //2. fold needs to pass in an initial value separately
    println("fold............")
    println(list.fold(10)(_ + _)) //10+1+2+3+4
    println(list.foldLeft(10)(_ - _)) // 10 - 1 - 2 - 3 - 4
    println(list.foldRight(11)(_ - _)) //3-(4-(5-(8-(10-11))))
  }
}

4) Case practice: merging two sets

package chapter07

import scala.collection.mutable

/**
 * Merge MAp
 */
object Test16_MergeMap {
  def main(args: Array[String]): Unit = {

    val map = Map("a" -> 1, "b" -> 3, "c" -> 6)
    val map2 = mutable.Map("a" -> 6, "b" -> 2, "c" -> 9, "d" -> 3)

    //Initial map2
    val map3: mutable.Map[String, Int] = map.foldLeft(map2)(
      //Update the value inside
      (mergedMap, kv) => {
      val key = kv._1
      val value = kv._2
      //mergedMap.getOrElse takes the current value + value
      mergedMap(key) = mergedMap.getOrElse(key, 0) + value
      mergedMap
    })

    println(map3)

  }
}

8, Common WordCount case

1) Demand

Word count: count the same words in the set and take the top three results

2) Demand analysis

3) Case practice

package chapter07

/**
 * Ordinary WordCount
 */
object Test17_CommonWordCount {

  def main(args: Array[String]): Unit = {

    val strings: List[String] = List(
      "hello",
      "hello word",
      "hello scala",
      "hello spark form scala",
      "hello flink from scala"
    )

    //1. The character pass is segmented to get a list of all the words
    val wordList1: List[Array[String]] = strings.map(_.split(" "))
    val wordList2: List[String] = wordList1.flatten
    println(wordList2)

    //2. Group the same words
    val groupMap: Map[String, List[String]] = wordList2.groupBy(word => word)
    println(groupMap)

    //3. Take the length of the LIST after grouping and get the number of each word
    val countMap: Map[String, Int] = groupMap.map(kv => (kv._1, kv._2.length))
    println("....countMap: " + countMap)

    //4. Convert the Map into a list and sort the top three
    val sortList: List[(String, Int)] = countMap.toList

      //The first one is_ 2 means to take the COUNT value of the second element and make a descending arrangement
      .sortWith(_._2 > _._2)
    println(sortList)

  }

}

9, Complex WordCount case

1) Mode 1

package chapter07

import scala.math.Ordering

object Test18_ComplexWordCount {
  def main(args: Array[String]): Unit = {
    val tupleList: List[(String, Int)] = List(
      ("hello", 1),
      ("hello word", 2),
      ("hello scala", 3),
      ("hello spark form scala", 1),
      ("hello flink from scala", 2))

    val newStringList: List[String] = tupleList.map(
      kv => {
        (kv._1.trim + " ") * kv._2
      })
    println(newStringList)


    //2. The next operation is exactly the same as the normal version
    val tuples: List[(String, Int)] = newStringList.flatMap(
      _.split(" "))
      .groupBy(e => e)
      .map(kv => (
        kv._1, kv._2.size
      ))
      .toList
      .sortBy(_._2)(Ordering[Int].reverse)
      .take(3)

    println(tuples)



    //Idea 2: convert directly based on the results of pre statistics
    println(".........Train of thought II...................")
    //1. Break up the string into words, and combine the corresponding number to wrap it into two tuples
    val preCountList: List[(String, Int)] = tupleList.flatMap(
      tuple => {
        val strings: Array[String] = tuple._1.split(" ")
        strings.map(word => (
          word, tuple._2
        ))
      }
    )
    println("......................."+preCountList)

    //2. Group two tuples according to words
    val preCountMap: Map[String, List[(String, Int)]] = preCountList.groupBy(tuple => tuple._1)
    println(preCountMap)


    //3. Superimpose the pre counted values of each word
    val countMap: Map[String, Int] = preCountMap.mapValues(
      tupleList => tupleList.map(_._2).sum
    )
    println("............"+countMap)

    //4. Convert to list and take the top three for sorting
    val countToList = countMap.toList
      .sortWith(_._2 > _._2)
      .take(3)
    println(countToList)
  }


}

10, Queue

1) Explain

Scala also provides the data structure of Queue, which is characterized by first in first out. Incoming and outgoing parties
The methods are enqueue and dequeue respectively.

2) Case practice

package chapter07

import scala.collection.immutable.Queue
import scala.collection.mutable

/**
 * Queue_ Queue
 */
object Test19_Queue {
  def main(args: Array[String]): Unit = {
    val queue: mutable.Queue[String] = new mutable.Queue[String]()

    //1. Create a variable queue
    queue.enqueue("a", "b", "c")
    println(queue)
    println(queue.dequeue())

    println(queue)
    println(queue.dequeue())

    println(queue)
    println(queue.dequeue())

    println("===========================")
    //Immutable queue
    val queue12: Queue[String] = Queue("a", "b", "c")
    val queue13 = queue12.enqueue("d")
    println(queue12)
    println(queue13)
  }
}

11, Parallel set

1) Explain

In order to make full use of multi-core CPU, Scala provides parallel set (different from the previous serial set) for multi-core CPU
Parallel computing in environment.

2) Case practice

package chapter07

import scala.collection.immutable

/**
 * Parallel set
 */
object Test20_Parallel {
  def main(args: Array[String]): Unit = {
    //serial 
    val strings: immutable.IndexedSeq[Long] = (1 to 100).map(
      x => Thread.currentThread().getId
    )
    println(strings)

    //parallel
    val result2 = (1 to 100).par.map(
      x => Thread.currentThread().getId
    )
    println(result2)
  }
}

Topics: Java Scala Big Data