Catalogue of series articles
Zltool kit source code learning notes (1) VS2019 source code compilation
Log function analysis of the tool module of zltool kit source code learning notes (2)
"Zltool kit source code learning notes" (3) terminal command analysis of tool module
Message broadcaster of tool module of zltool kit source code learning notes (4)
Resource pool of tool module of zltool kit source code learning notes (5)
Zltool kit source code learning notes (6) overview of the overall framework of the threading module
Zltool kit source code learning notes (8) thread load calculator of thread module
Zltool kit source code learning notes (9) task executor of thread module
Zltool kit source code learning notes (10) thread pool of thread module
Learning notes on ZLToolKit source code (11) working thread pool of thread module
Learning notes on zltool kit source code (14) timer of event polling module
Learning notes on zltool kit source code (15) event poller of event polling module (this article)
preface
A server program generally needs to handle three types of events: IO events, timing events and signals. In order to facilitate processing, we need to unify the event source, such as using IO replication to manage all events. Secondly, in order to realize cross platform, we need to provide a platform independent unified interface, and the platform related implementation is completed internally. For example, for IO reuse, epoll is available under linux, and select is available on other platforms such as windows. Here, we can uniformly package and provide consistent interfaces. An event management interface should also support multithreading. In this section, learn the event management tool in zltool Kit: EventPoller. It supports timing event and IO event processing. Through the built-in pipeline event, it can also realize the load balancing of multithreaded tasks.
catalogue
2.1. 2. onPipeEvent - Pipeline read event callback
2.1. 3,async_l - add tasks to be executed in the event listening thread
2.3. 1. addEvent - add event listener
1, Overview
According to my understanding, the interfaces of EventPoller class are roughly divided into four categories: internal pipeline events, timer events, user events (events that can be managed by users, such as pipelines, network socket s, etc.) and thread related interfaces.
The event processing mode of EventPoller is Reactor mode. Each event has its corresponding callback. After the event is triggered, the callback is called.
The event management of EventPoller is generally carried out in a separate thread, which can be determined by the runLoop function. The first parameter of the interface can set whether it is blocked. In the non blocking state, a thread will be created, and subsequent event listening will be carried out in the thread. In linux system, event monitoring is managed through epoll, and in non linux platform, it is managed through select.
Although this class is a singleton class (constructor private), because it takes EventPollerPool as a friend class, we generally do not instantiate this class object directly, but use it indirectly through its friend class EventPollerPool. In the EventPollerPool, multiple instances of EventPoller will be created according to the size specified by the user or the number of CPU cores. During subsequent use, select one of them according to specific conditions (for example, select the one with the lightest load).
For the learning of EventPollerPool, see Learning notes on ZLToolKit source code (11) working thread pool of thread module , the two are similar.
2, Functional analysis
2.1 internal pipeline events
Internal pipeline events are used for communication between user worker threads and event listener threads. In the event listening thread, the read end of the pipeline is monitored. In the user working thread, the write end of the pipeline is used.
Based on the pipeline event, a task queue is implemented, which can execute tasks asynchronously in the thread_ list_task is used to store the tasks associated with the pipeline, async_l be responsible for adding tasks to_ list_ In task, the data is written to the pipeline, triggering its read event. After monitoring the event in runLoop, the onPipeEvent is called to execute and clear. list_ Tasks in task.
2.1. 1. Constructor
EventPoller::EventPoller(ThreadPool::Priority priority ) {
_priority = priority;
SockUtil::setNoBlocked(_pipe.readFD());
SockUtil::setNoBlocked(_pipe.writeFD());
#if defined(HAS_EPOLL)
_epoll_fd = epoll_create(EPOLL_SIZE);
if (_epoll_fd == -1) {
throw runtime_error(StrPrinter << "establish epoll File descriptor failed:" << get_uv_errmsg());
}
SockUtil::setCloExec(_epoll_fd);
#endif //HAS_EPOLL
_logger = Logger::Instance().shared_from_this();
_loop_thread_id = this_thread::get_id();
//Add internal pipeline event
if (addEvent(_pipe.readFD(), Event_Read, [this](int event) { onPipeEvent(); }) == -1) {
throw std::runtime_error("epoll Failed to add pipeline");
}
}As you can see, the reading end fd of the pipeline is added to the event list. When the pipeline reading event is triggered, the onPipeEvent function will be executed.
2.1. 2. onPipeEvent - Pipeline read event callback
inline void EventPoller::onPipeEvent() {
TimeTicker();
char buf[1024];
int err = 0;
do {
if (_pipe.read(buf, sizeof(buf)) > 0) {
continue;
}
err = get_uv_error(true);
} while (err != UV_EAGAIN);
decltype(_list_task) _list_swap;
{
lock_guard<mutex> lck(_mtx_task);
_list_swap.swap(_list_task);
}
_list_swap.for_each([&](const Task::Ptr &task) {
try {
(*task)();
} catch (ExitException &) {
_exit_flag = true;
} catch (std::exception &ex) {
ErrorL << "EventPoller Exception caught while executing asynchronous task:" << ex.what();
}
});
}This interface is called and executed after the pipeline read event occurs_ list_ Tasks in task. First read the data in the pipeline, and then exchange_ list_task, that is, clear the tasks in it. Finally, perform all tasks in this way.
2.1. 3,async_l - add tasks to be executed in the event listening thread
Task::Ptr EventPoller::async_l(TaskIn task,bool may_sync, bool first) {
TimeTicker();
if (may_sync && isCurrentThread()) {
task();
return nullptr;
}
auto ret = std::make_shared<Task>(std::move(task));
{
lock_guard<mutex> lck(_mtx_task);
if (first) {
_list_task.emplace_front(ret);
} else {
_list_task.emplace_back(ret);
}
}
//Write data to the pipeline and wake up the main thread
_pipe.write("", 1);
return ret;
}In the case of asynchronous execution, add the task to_ list_ In task, the event is triggered by writing data to the pipeline. In the runLoop event listener thread, the onPipeEvent is called to perform the task.
2.2 timing events
This part has been learned in the timer section. See Learning notes on zltool kit source code (14) timer of event polling module.
2.3. User events
Users can let EventPoller manage the events they care about through three interfaces: addEvent, delEvent and modifyEvent. All managed events are stored in_ event_map.
unordered_map<int, Poll_Record::Ptr> _event_map;
2.3. 1. addEvent - add event listener
int EventPoller::addEvent(int fd, int event, PollEventCB cb) {
TimeTicker();
if (!cb) {
WarnL << "PollEventCB Empty!";
return -1;
}
if (isCurrentThread()) {
#if defined(HAS_EPOLL)
struct epoll_event ev = {0};
ev.events = (toEpoll(event)) | EPOLLEXCLUSIVE;
ev.data.fd = fd;
int ret = epoll_ctl(_epoll_fd, EPOLL_CTL_ADD, fd, &ev);
if (ret == 0) {
_event_map.emplace(fd, std::make_shared<PollEventCB>(std::move(cb)));
}
return ret;
#else
#ifndef _WIN32
//On win32 platform, socket socket is not equal to file descriptor, so this restriction may not apply
if (fd >= FD_SETSIZE || _event_map.size() >= FD_SETSIZE) {
WarnL << "select Maximum listening" << FD_SETSIZE << "File descriptors";
return -1;
}
#endif
Poll_Record::Ptr record(new Poll_Record);
record->event = event;
record->callBack = std::move(cb);
_event_map.emplace(fd, record);
return 0;
#endif //HAS_EPOLL
}
async([this, fd, event, cb]() {
addEvent(fd, event, std::move(const_cast<PollEventCB &>(cb)));
});
return 0;
}Register the event in epoll or select for management. Pay attention to the async call at the end of the function (if (isCurrentThread()) in front determines whether it is an event listening thread). It shows that the actual operation of the interface is not executed in the user's working thread, but transferred to the event listening thread, avoiding the multi thread operation epoll or select.
Each event has its corresponding callback, which is called after the event is triggered.
For epoll, you can directly call epoll series APIs to operate. For select, you need to re add the event to listen to each time, so only put the event in the_ event_ In map, the operations added to select are executed in runLoop. The relevant codes in the excerpt runLoop are as follows:
set_read.fdZero();
set_write.fdZero();
set_err.fdZero();
max_fd = 0;
for (auto &pr : _event_map) {
if (pr.first > max_fd) {
max_fd = pr.first;
}
if (pr.second->event & Event_Read) {
set_read.fdSet(pr.first);//Listen for pipeline readable events
}
if (pr.second->event & Event_Write) {
set_write.fdSet(pr.first);//Listen for pipeline writable events
}
if (pr.second->event & Event_Error) {
set_err.fdSet(pr.first);//Listen for pipeline error events
}
}
startSleep();//Used to count the current thread load
ret = zl_select(max_fd + 1, &set_read, &set_write, &set_err, minDelay ? &tv : NULL);
sleepWakeUp();//Used to count the current thread loaddelEvent, modifyEvent and addEvent interfaces are similar and will not be repeated.
3, runLoop
Internal pipeline events, timer events and user added events are managed in this interface. When the event is triggered, call their corresponding callback functions to notify the event. When performing IO operations and other time-consuming work synchronously in the callback, the event cannot be processed in time. This operation should be avoided as far as possible. You can use the task queue to put the ready event in the task queue, and then another worker thread obtains the task execution from the queue.