Guiding ideology:
1. Using RPC communication framework (AKKA)
2. Define 2 classes Master and Worker
-------------------------------------------------------------------------------------------------------------------------------
Start Master first, then all workers
1. After the Worker is started, establish a connection with the Master in the PreStart method, send the registration to the Master, and encapsulate the Worker's information to the Master through the case class.
2. The Master receives the registration message of the Worker, saves the information of the Worker, and then feeds back the success of the registration to the Worker.
3. The Worker sends heartbeat to the Master on a regular basis to report the survival status.
4. Master will periodically clean up the overtime workers.
--------------------------------------------------------------------------------------------------------------------------------
First, open the Idea software and create the maven project:
Write Pom file, the code of Pom file is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cn.allengao.akka</groupId>
<artifactId>my-rpc</artifactId>
<version>1.0</version>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.10.6</scala.version>
<scala.compat.version>2.10</scala.compat.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.10</artifactId>
<version>2.3.14</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-remote_2.10</artifactId>
<version>2.3.14</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-make:transitive</arg>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>cn.allengao.rpc.Worker</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>The code of RemoteMessage is as follows:
package cn.allengao.rpc /** * class_name: * package: * describe: TODO * creat_user: Allen Gao * creat_date: 2018/1/18 * creat_time: 16:36 **/ trait RemoteMessage extends Serializable //Worker -> Master case class RegisterWorker(id: String, host: String, port: Int, memory: Int, cores: Int) extends RemoteMessage case class Heartbeat(workId: String) extends RemoteMessage //Master -> Worker case class RegisteredWorker(masterUrl: String) extends RemoteMessage //Worker -> self case object SendHeartbeat // Master -> self case object CheckTimeOutWorker
The code of WorkerInfo is as follows:
package cn.allengao.rpc
/**
* class_name:
* package:
* describe: Encapsulate the information of worker startup and pass it to the master.
* creat_user: Allen Gao
* creat_date: 2018/1/18
* creat_time: 16:37
**/
class WorkerInfo(val id: String, val host:String, val port: Int, val memory: Int, val cores: Int) {
//TODO last heartbeat
var lastHeartbeatTime : Long = _
}
The code of Master is as follows:
package cn.allengao.rpc
import akka.actor.{Actor, ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import scala.collection.mutable
import scala.concurrent.duration._
/**
* class_name:
* package:
* describe: TODO
* creat_user: Allen Gao
* creat_date: 2018/1/18
* creat_time: 16:36
**/
class Master(val host: String, val port: Int) extends Actor {
// workerId -> WorkerInfo
val idToWorker = new mutable.HashMap[String, WorkerInfo]()
// WorkerInfo
val workers = new mutable.HashSet[WorkerInfo]() //Use set to delete fast or linkList
//Interval of timeout check
val CHECK_INTERVAL = 15000
//The preStart method is executed after the constructor executes the constructor and before the receive method, and only once.
override def preStart(): Unit = {
println("preStart invoked")
//Import implicit conversion, start a timer in preStart, which is used to check the overtime Worker of cycle
import context.dispatcher //It's too low to use timer. You can use akka's, use timer to import this package
context.system.scheduler.schedule(0 millis, CHECK_INTERVAL millis, self, CheckTimeOutWorker)
}
// For receiving messages, the receive method loops.
override def receive: Receive = {
case RegisterWorker(id, host, port, memory, cores) => {
//Check to see if you have already registered
if(!idToWorker.contains(id)){
//Encapsulate the Worker's information and save it in memory
val workerInfo = new WorkerInfo(id, host, port, memory, cores)
idToWorker += (id -> workerInfo) //key is id, value is workInfo
workers += workerInfo
println("a worker registered")
// sender ! RegisteredWorker(s"akka.tcp://MasterSystem@$host:$port/user/Master")
sender ! RegisteredWorker(s"akka.tcp://${Master.MASTER_SYSTEM}" +
s"@$host:$port/user/${Master.MASTER_ACTOR}")//Notify worker to register
}
}
case Heartbeat(workerId) => {
if(idToWorker.contains(workerId)){
val workerInfo = idToWorker(workerId)
//Newspaper activity
val current_time = System.currentTimeMillis()
workerInfo.lastHeartbeatTime = current_time
}
}
case CheckTimeOutWorker => {
val currentTime = System.currentTimeMillis()
val toRemove : mutable.HashSet[WorkerInfo] = workers.filter(w => currentTime - w.lastHeartbeatTime > CHECK_INTERVAL)
// for(w <- toRemove) {
// workers -= w
// idToWorker -= w.id
// }
toRemove.foreach(deadWorker =>{
idToWorker -= deadWorker.id
workers -= deadWorker
})
println("num of workers " + workers.size)
}
}
}
object Master {
//Declare two variables MaterSystem and MasterActor
val MASTER_SYSTEM = "MasterSystem"
val MASTER_ACTOR = "Master"
def main(args: Array[String]) {
//Resolve the incoming parameter hostname and port number
val host = args(0)
val port = args(1).toInt
// Prepare the configuration information: (| between is the parsing data, divided by "=", before "=" is the parameter (key), after which is the value.)
val configStr =
s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname = "$host"
|akka.remote.netty.tcp.port = "$port"
""".stripMargin
//ConfigFactory profile information class passes the profile information in
val config = ConfigFactory.parseString(configStr)
//The leader of Actor system, who assists in creating and monitoring the following actors, is an example
val actorSystem = ActorSystem(MASTER_SYSTEM, config)
//Create Actor
val master = actorSystem.actorOf(Props(new Master(host, port)), MASTER_ACTOR)
//Call thread wait
actorSystem.awaitTermination()
}
}
The code of Worker is as follows:
package cn.allengao.rpc
/**
* class_name:
* package:
* describe: TODO
* creat_user: Allen Gao
* creat_date: 2018/1/18
* creat_time: 16:37
**/
import java.util.UUID
import akka.actor.{Actor, ActorSelection, ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration._
/**
* Created by root on 2016/5/13.
*/
class Worker(val host: String, val port: Int,val masterHost: String,
val masterPort: Int, val memory: Int, val cores: Int) extends Actor{
val worker_id = UUID.randomUUID().toString
var masterUrl : String = _
val HEART_INTERVAL = 10000
var master : ActorSelection = _
//
override def preStart(): Unit = {
//Establish connection with Master
// master = context.actorSelection(s"akka.tcp://MasterSystem@$masterHost:$masterPort/user/Master")
master = context.actorSelection(s"akka.tcp://${Master.MASTER_SYSTEM}" +
s"@$masterHost:$masterPort/user/${Master.MASTER_ACTOR}")
//Send registration message to Master
master ! RegisterWorker(worker_id,host,port, memory, cores)
}
override def receive: Receive = {
case RegisteredWorker(masterUrl) => {
println(masterUrl)
//Start timer to send heartbeat, heartbeat is a case class
//Import an implicit conversion to start the timer
import context.dispatcher
//How long does it take to execute the unit? How often does it take to execute the unit? The receiver of the message
// (it's not good to send messages directly to the master. You can send messages to yourself first, and then you can judge when to send messages)
context.system.scheduler.schedule(0 millis, HEART_INTERVAL millis, self, SendHeartbeat)
}
case SendHeartbeat => {
println("send heartbeat to master")
//Check before sending the heartbeat
master ! Heartbeat(worker_id)
}
}
}
object Worker {
val WORKER_SYSTEM = "WorkerSystem"
val WORKER_ACTOR = "Worker"
def main(args: Array[String]) {
val host = args(0)
val port = args(1).toInt
val masterHost = args(2)
val masterPort = args(3).toInt
//Memory and CPU cores used for analysis tasks
val memory = args(4).toInt
val cores = args(5).toInt
// Prepare configuration
val configStr =
s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname = "$host"
|akka.remote.netty.tcp.port = "$port"
""".stripMargin
val config = ConfigFactory.parseString(configStr)
//The leader of Actor system, who assists in creating and monitoring the following actors, is a single example
val actorSystem = ActorSystem(WORKER_SYSTEM, config)
actorSystem.actorOf(Props(new Worker(host, port, masterHost, masterPort, memory, cores)), WORKER_ACTOR)
//Call thread wait
actorSystem.awaitTermination()
}
} Run the code of Master: simulate the parameters passed in as follows
Then right click Run Master, the effect is as follows:
Run Woker's program: simulate the parameters passed in as follows
Right click the code to run the Worker, and the effect is as follows:
When the worker is stopped, it can be seen from the running status of the master that the worker connection is detected to be disconnected, and the number of connections returns to "0".
So far, the RPC communication small instance of the whole akka has been tested.