2021SC@SDUSC
1, Introduction to RPC
As a distributed system, the design of HBase is a typical master save architecture. HBase mainly has three roles: Master, RegionServer and Client, and RPC is the communication link between master, RegionServer and Client.
Client
There are many clients, such as hbase shell, java client API, etc. the client does not provide any RPC services, but only calls the services provided by RegionServer or Master.
Master
Master mainly implements MasterService and RegionServerStatus protocols, which are called by Client and RegionServer respectively.
MaterService
MasterService mainly defines some DML related services such as obtaining cluster status, obtaining meta information of tables, adding / deleting columns, assign region, enable/disable table, load balancing, etc. The Master provides the implementation of these services for the client to call. For example, when we run enable/disable table and other related commands in the hbase shell, the client will first send the RPC request to the Master.
RegionServerStatus
RegionServerStatus mainly defines that the RegionServer reports the cluster status to the Master. The RegionServer starts sending RPC requests and other related services to the Master, and the Master can understand the status of the RegionServer in the whole cluster according to these RPC request information.
ReginServer
RegionServer mainly implements the AdminService and ClientService protocols, which can be called by the client side. At the same time, ReginServer will also call the RegionServerStatus service to report relevant information to the master.
AdminService
AmdinService mainly defines services related to obtaining table region information and operating region (Open,Flush,Split, Compact, Merge, etc.).
ClientService
ClientService is mainly used to obtain data, update, add data, Scan and other related services.
2, Overview of RPC in HBase
RPC (remote procedure call) is a remote procedure call. For local calls, after a function is defined, other parts of the program can return the desired results by calling the function. The only difference between RPC is that function definitions and function calls are usually located on different machines. Because different machines are involved, RPC has more communication parts than local function calls, mainly involving two role callers (Client side) and function definition implementation (Server side).
Server side RPC implementation
1.RPC initialization
public HRegionServer(final Configuration conf) throws IOException {
super("RegionServer"); // thread name
TraceUtil.initTracer(conf);
//...
rpcServices = createRpcServices();
//...
In the source code of HRegionServer startup class, there is the above code to initialize RPCServer.
private void preRegistrationInitialization() {
//...
this.rpcClient = RpcClientFactory.createClient(conf, clusterId, new InetSocketAddress(
this.rpcServices.isa.getAddress(), 0), clusterConnection.getConnectionMetrics());
//...
}
In the source code of HRegionServer startup class, there is the above code to initialize RpcClient.
public MasterRpcServices(HMaster m) throws IOException {
super(m);
master = m;
}
Create the construction method of HMaster and MasterRpcServices and call the construction method of parent class RSRpcServices
RSRpcServices(final HRegionServer rs, final LogDelegate ld) throws IOException {
final Configuration conf = rs.getConfiguration();
this.ld = ld;
regionServer = rs;
//...
final RpcSchedulerFactory rpcSchedulerFactory;
try {
rpcSchedulerFactory = getRpcSchedulerFactoryClass().asSubclass(RpcSchedulerFactory.class)
.getDeclaredConstructor().newInstance();
} catch (NoSuchMethodException |
// Creation of a HSA will force a resolve.
initialIsa = new InetSocketAddress(hostname, port);
bindAddress =
new InetSocketAddress(conf.get("hbase.regionserver.ipc.address", hostname), port);
}
//...
priority = createPriority();
//...
ConnectionUtils.setServerSideHConnectionRetriesConfig(conf, name, LOG);
rpcServer = createRpcServer(rs, rpcSchedulerFactory, bindAddress, name);
rpcServer.setRsRpcServices(this);
//...
Set HRegionServer through regionServer=rs;
Next, RpcSchedulerFactory rpcSchedulerFactory =... Initialize RpcSchedurFactory, reflect the class specified by hbase.region.server.rpc.scheduler.factory.class, and use SimpleRpcSchedulerFactory by default;
When scheduling requests, use the priority=createPriority() priority function and set the number of retries during access through connectionutils.setserversidehconnectionretriesconfig
public SimpleRpcServer(final Server server, final String name,
final List<BlockingServiceAndInterface> services,
final InetSocketAddress bindAddress, Configuration conf,
RpcScheduler scheduler, boolean reservoirEnabled) throws IOException {
super(server, name, services, bindAddress, conf, scheduler, reservoirEnabled);
this.socketSendBufferSize = 0;
this.readThreads = conf.getInt("hbase.ipc.server.read.threadpool.size", 10);
this.purgeTimeout = conf.getLong("hbase.ipc.client.call.purge.timeout",
2 * HConstants.DEFAULT_HBASE_RPC_TIMEOUT);
// Start the listener here and let it bind to the port
listener = new Listener(name);
this.port = listener.getAddress().getPort();
// Create the responder here
responder = new SimpleRpcServerResponder(this);
connectionManager = new ConnectionManager();
initReconfigurable(conf);
this.scheduler.init(new RpcSchedulerContext(this));
}
This constructor is the RpcServerInterface interface implemented by RpcServer to set the Listener and Responder. At the same time, the caller creates and sets the Scheduler through RpcSchedulerFactory.
2,Listener
The Listener is responsible for listening to requests. For the obtained requests, the reader is responsible for reading them.
private class Listener extends Thread {
private ServerSocketChannel acceptChannel = null; //the accept channel
private Selector selector = null; //the selector that we use for the server
private Reader[] readers = null;
private int currentReader = 0;
private final int readerPendingConnectionQueueLength;
private ExecutorService readPool;
public Listener(final String name) throws IOException {
super(name);
//...
acceptChannel = ServerSocketChannel.open();
acceptChannel.configureBlocking(false);
// Bind the server socket to the binding addrees (can be different from the default interface)
bind(acceptChannel.socket(), bindAddress, backlogLength);
port = acceptChannel.socket().getLocalPort(); //Could be an ephemeral port
address = (InetSocketAddress)acceptChannel.socket().getLocalSocketAddress();
// create a selector;
selector = Selector.open();
readers = new Reader[readThreads];
readPool = Executors.newFixedThreadPool(readThreads,
new ThreadFactoryBuilder().setNameFormat(
"Reader=%d,bindAddress=" + bindAddress.getHostName() +
",port=" + port).setDaemon(true)
.setUncaughtExceptionHandler(Threads.LOGGING_EXCEPTION_HANDLER).build());
for (int i = 0; i < readThreads; ++i) {
Reader reader = new Reader();
readers[i] = reader;
readPool.execute(reader);
}
//...
acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
this.setName("Listener,port=" + port);
this.setDaemon(true);
}
Create a non blocking ServerSocketChannel through acceptChannel=ServerSocketChannel.open();
bind(acceptChannel.socket(),bindAddress,backlogLength) bind the socket to RpcServer#bingAddress;
Next, create a selector, initialize the Reader ThreadPool with readers=new Reader[readThreads], and complete the registration of the selector.
Listener listens to op_ The accept and doaccept methods are to select a reader and register the op of the channel returned by accept_ Read event, and construct a Connection object for read to get.
3,Reader
The logic for processing the request is in the Reader, and the Call object is generated and handed over to the RPCSchedule for distribution.
private class Reader implements Runnable {
final private LinkedBlockingQueue<SimpleServerRpcConnection> pendingConnections;
private final Selector readSelector;
public void run() {
try {
doRunLoop();
} finally {
try {
readSelector.close();
} catch (IOException ioe) {
LOG.error(getName() + ": error closing read selector in " + getName(), ioe);
}
}
}
private synchronized void doRunLoop() {
while (running) {
try {
int size = pendingConnections.size();
for (int i=size; i>0; i--) {
SimpleServerRpcConnection conn = pendingConnections.take();
conn.channel.register(readSelector, SelectionKey.OP_READ, conn);
}
readSelector.select();
Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
while (iter.hasNext()) {
SelectionKey key = iter.next();
iter.remove();
if (key.isValid()) {
if (key.isReadable()) {
doRead(key);
}
}
key = null;
}
}
}
}
In synchronized, if the thread is blocked, it knows that a request has arrived; If the request is valid, doRead(key) performs subsequent processing.
void doRead(SelectionKey key) throws InterruptedException {
int count;
SimpleServerRpcConnection c = (SimpleServerRpcConnection) key.attachment();
try {
count = c.readAndProcess();
} catch (InterruptedException ieo) {
LOG.info(Thread.currentThread().getName() + ": readAndProcess caught InterruptedException", ieo);
throw ieo;
} catch (Exception e) {
//...
}
}
if (!this.rpcServer.scheduler.dispatch(new CallRunner(this.rpcServer, call))) {
this.rpcServer.callQueueSizeInBytes.add(-1 * call.getSize());
//...
call.sendResponseIfReady();
}
}
reader listening op_ Read and doread () methods are handled by the Connection object in the Listener, generate a Call, wrap it as a callrunner and give it to the Scheduler. Read the byte stream data from the Connection and process the request (construct the RequestHeader, method, parameter and callrunner object, which will be distributed by the Scheduler).
4,Scheduler
Scheduler is a producer consumer model. There is a queue to cache requests, and some threads are responsible for pulling messages from the queue for distribution.
public class SimpleRpcScheduler extends RpcScheduler implements ConfigurationObserver {
private int port;
private final PriorityFunction priority;
private final RpcExecutor callExecutor;
private final RpcExecutor priorityExecutor;
private final RpcExecutor replicationExecutor;
In the process of server implementation, hbase rpc implements two schedulers, FifoRPCScheduler and SimpleRpcScheduler.
Fiforpcs scheduler will directly put the CallRunner object into the thread pool for execution, while simplerpcs scheduler will be divided into three different executors. For different requests, different executors will be used for execution.
The default implementation of the Scheduler is SimpleRpcScheduler, which contains three rpceexecutors (callExecutor, priorityexecution and replicationexecution). Different executors are used to execute different requests
if (null != callExecutor) {
queueName = "Call Queue";
callQueueInfo.setCallMethodCount(queueName, callExecutor.getCallQueueCountsSummary());
callQueueInfo.setCallMethodSize(queueName, callExecutor.getCallQueueSizeSummary());
}
if (null != priorityExecutor) {
queueName = "Priority Queue";
callQueueInfo.setCallMethodCount(queueName, priorityExecutor.getCallQueueCountsSummary());
callQueueInfo.setCallMethodSize(queueName, priorityExecutor.getCallQueueSizeSummary());
}
if (null != replicationExecutor) {
queueName = "Replication Queue";
callQueueInfo.setCallMethodCount(queueName, replicationExecutor.getCallQueueCountsSummary());
callQueueInfo.setCallMethodSize(queueName, replicationExecutor.getCallQueueSizeSummary());
}
Select different executors to process distribution requests. Most of the requests based on are executed through callExecutor.
protected List<BlockingQueue<CallRunner>> getQueues() {
return queues;
}
protected void startHandlers(final int port) {
List<BlockingQueue<CallRunner>> callQueues = getQueues();
startHandlers(null, handlerCount, callQueues, 0, callQueues.size(), port, activeHandlerCount);
}
The implementation class rwqueuerpceexecutor of rpceexecutor caches messages using blocking queues. After scheduler scheduling, CallRunner calls the Call method in RpcServer, and callBlockingMethod calls the function of server to implement the service and returns the result.
5,Responder
The responder is responsible for sending the RPC request result to the Client. After the Scheduler schedules the request, the execution result is added to the corresponding queue of the returned result through doRespond().
void doRespond(SimpleServerRpcConnection conn, RpcResponse resp) throws IOException {
if (conn.responseQueue.isEmpty() && conn.responseWriteLock.tryLock()) {
try {
if (conn.responseQueue.isEmpty()) {
// If we're alone, we can try to do a direct call to the socket. It's
// an optimization to save on context switches and data transfer between cores..
if (processResponse(conn, resp)) {
return; // we're done.
}
// Too big to fit, putting ahead.
conn.responseQueue.addFirst(resp);
added = true; // We will register to the selector later, outside of the lock.
}
} finally {
conn.responseWriteLock.unlock();
}
}
if (conn.responseQueue.isEmpty() && conn.responseWriteLock.tryLock()) {
try {
if (conn.responseQueue.isEmpty()) {
if (processResponse(conn, resp)) {
return; // we're done.
}
conn.responseQueue.addFirst(resp);
added = true; // We will register to the selector later, outside of the lock.
}
} finally {
conn.responseWriteLock.unlock();
}
}
if (!added) {
conn.responseQueue.addLast(resp);
}
registerForWrite(conn);
}
}
RpcServer: after the call returns the result, wrap the returned result response, push the current call into the responseQueue through the doResponse function, and write the current connectionResponder to set: writingcon.
private void registerWrites() {
Iterator<SimpleServerRpcConnection> it = writingCons.iterator();
//...
private void doRunLoop() {
while (this.simpleRpcServer.running) {
try {
registerWrites();
int keyCt = writeSelector.select(this.simpleRpcServer.purgeTimeout);
if (keyCt == 0) {
continue;
}
Set<SelectionKey> keys = writeSelector.selectedKeys();
Iterator<SelectionKey> iter = keys.iterator();
while (iter.hasNext()) {
SelectionKey key = iter.next();
iter.remove();
try {
if (key.isValid() && key.isWritable()) {
doAsyncWrite(key);
}
} catch (IOException e) {
SimpleRpcServer.LOG.debug(getName() + ": asyncWrite", e);
}
}
//...
private boolean processAllResponses(final Connection connection) throws IOException {
connection.responseWriteLock.lock();
try {
for (int i = 0; i < 20; i++) {
Call call = connection.responseQueue.pollFirst();
if (!processResponse(call)) {
connection.responseQueue.addFirst(call);
return false;
}
}
} finally {
connection.responseWriteLock.unlock();
}
return connection.responseQueue.isEmpty();
}
}
If the write operation is not completed in doRespond(), the thread in the Responder performs subsequent operations by registering the connection of the Call object with the selector.
summary
Steps of rpc implementation on server side
1. The implementation logic of the server side is mainly encapsulated in the rpcserver object. In this object, there must be a listener object to monitor the Connection request. Once there is a Connection, the listener will select a Reader and register op on the new Connection_ The read event encapsulates the Connection object.
2. The Reader first reads data from the connection, and finally constructs a callRunner, which is called by the scheduler. There are usually multiple Reader objects in the rpcserver object.
3. Call callrunner: Run - > rpcserver: call according to a callrunner object selected by the scheduler to call the implementation of specific functions.
4. After the function returns the result, it is wrapped into a response object, and the current call is pushed into the responseQueue through the doReponse function, and the current conResponder needs to be written to writingcon.
5. Register the op connected in writingCons_ Write event, get the call from the responseQueue, process it, and send the byte stream of the result.