Original: www.bluefoxah.org/teamtalk/server_flow.html

Preface

In the previous article, we simply analyzed the configuration of each server. In this article, we simply analyzed the whole operation process of TeamTalk server.

Server-side process

There is no strict sequential process for the start-up of the server, because each end will actively connect to the server on which it depends after the start-up. However, in this case, if it is an online environment, it is recommended to start in the following order (not the only order):
1. Start db_proxy.
2. Start route_server, file_server, msfs
3. Start login_server
4. Start msg_server

Then I will explain a process overview of the server according to the start-up sequence of the server.
Step 1: After starting db_proxy, db_proxy will connect to the corresponding configuration file. MySQL Examples, and Redis Example.
Step 2: After starting route_server, file_server and msfs, each server will start to listen on the corresponding ports.
Step 3: Start login_server,login_server starts to listen on the corresponding port, waiting for the client to connect, and assign a relatively small load msg_server to the client.
Step 4: Start the msg_server. When the msg_server is started, it will actively connect route_server, login_server, db_proxy. It will register the port information it monitors to login_server. At the same time, when the user is online and offline, it will report its load to login_server.
Next, I will analyze some logic of route_server, login_server and msg_server. Here is a simple analysis, followed by specific analysis for each end.
Firstly, the structure of each end is explained.

route_server

The role of route_server in the whole tt is a place for message forwarding, which maintains global user information in memory. When there are multiple msg_servers, route_server is used to transfer messages between multiple msg_servers.
In RouteConn.cpp, the following structure is defined to save user status:

1. typedef map<CRouteConn*, uint32_t> RouteConnMap_t;  
2. typedef struct {  
3.     uint32_t    status;  
4.     RouteConnMap_t      conns;  
5. } UserStat_t;  
6.   
7. typedef hash_map<uint32_t, UserStat_t> UserStatMap_t;  
8.   
9. static UserStatMap_t g_rs_user_map;  

typedef map<CRouteConn*, uint32_t> RouteConnMap_t;
typedef struct {
    uint32_t    status;
    RouteConnMap_t      conns;
} UserStat_t;

typedef hash_map<uint32_t, UserStat_t> UserStatMap_t;

static UserStatMap_t g_rs_user_map;

g_rs_user_map is a hash_map that stores global user information. It is defined as follows: key is a user id, value is a structure:

1. typedef struct {  
2.     uint32_t    status;  
3.     RouteConnMap_t      conns;  
4. } UserStat_t;  

typedef struct {
    uint32_t    status;
    RouteConnMap_t      conns;
} UserStat_t;

In this structure, status identifies the user's status, and conns is also a map, which stores the corresponding msg_server connection and whether the user is on the pc or mobile side of the connection.

When a user is online again, msg_server sends the user's status to route_server, and route_server inserts a record in g_rs_user_map.

login_server

login_server in the entire TT Framework In login_server, the address of all msg_server s and their current load situation are also maintained in memory.
In LoginConn.cpp, the following structure is defined to save the state machine load of msg_server:

1. typedef hash_map<uint32_t, uint32_t> UserConnCntMap_t;  
2. typedef struct  {  
3.     string      ip_addr1;   //Netcom IP  
4.     string      ip_addr2;   //Telecom IP  
5.     uint16_t    port;  
6.     uint32_t    max_conn_cnt;  
7.     uint32_t    cur_conn_cnt;  
8.     uint32_t    cur_user_cnt;   //Current number of users - cur_user_cnt!= cur_conn_cnt because users are allowed to log in at multiple points  
9.     string      hostname;   //Host name of message server  
10.     uint32_t    server_type;  
11.     UserConnCntMap_t user_cnt_map;  
12. } msg_serv_info_t;  
13.   
14. static map<uint32_t, msg_serv_info_t*> g_msg_serv_info;  

typedef hash_map<uint32_t, uint32_t> UserConnCntMap_t;
typedef struct  {
    string      ip_addr1;   // Netcom IP
    string      ip_addr2;   // Telecom IP
    uint16_t    port;
    uint32_t    max_conn_cnt;
    uint32_t    cur_conn_cnt;
    uint32_t    cur_user_cnt;   // Current User Number - cur_user_cnt!= cur_conn_cnt because users are allowed to log in at multiple points
    string      hostname;   // Host name of message server
    uint32_t    server_type;
    UserConnCntMap_t user_cnt_map;
} msg_serv_info_t;

static map<uint32_t, msg_serv_info_t*> g_msg_serv_info;

g_msg_serv_info is a hash_map used to save msg_server. The key is a socket descriptor and the value is a structure that contains specific information about the corresponding msg_server:

1. typedef struct  {  
2.     string      ip_addr1;   //Netcom IP  
3.     string      ip_addr2;   //Telecom IP  
4.     uint16_t    port;  
5.     uint32_t    max_conn_cnt;  
6.     uint32_t    cur_conn_cnt;  
7.     uint32_t    cur_user_cnt;   //Current number of users - cur_user_cnt!= cur_conn_cnt because users are allowed to log in at multiple points  
8.     string      hostname;   //Host name of message server  
9.     uint32_t    server_type;  
10.     UserConnCntMap_t user_cnt_map;  
11. } msg_serv_info_t;  

typedef struct  {
    string      ip_addr1;   // Netcom IP
    string      ip_addr2;   // Telecom IP
    uint16_t    port;
    uint32_t    max_conn_cnt;
    uint32_t    cur_conn_cnt;
    uint32_t    cur_user_cnt;   // Current User Number - cur_user_cnt!= cur_conn_cnt because users are allowed to log in at multiple points
    string      hostname;   // Host name of message server
    uint32_t    server_type;
    UserConnCntMap_t user_cnt_map;
} msg_serv_info_t;

ip_addr1 and ip_addr2 respectively store the IP of communication and telecommunication corresponding to msg_server (which is designed here because our server is two-wire). Port saves the port corresponding to msg_server. max_conn_cnt stores the maximum number of connections configured in msg_server. When the number of connections exceeds this value, msg_server does not score. In the past, cur_conn_cnt corresponded to the current number of connections of msg_server. cur_user_cnt saved the number of users corresponding to msg_server login. As the commentary said, the current number of users <= the current number of connections, hostname saved the hostname of msg_server, server_type labeled msg_server. Network type, currently only one definition of Tcp Server, defined in ImPduServer.h:

1. enum {  
2.     MSG_SERVER_TYPE_TCP     = 1,  
3. };  

enum {
    MSG_SERVER_TYPE_TCP     = 1,
};

user_cnt_map is a hash_map used to save the user id and the corresponding number of users:

1. hash_map<uint32_t, uint32_t> UserConnCntMap_t;  

hash_map<uint32_t, uint32_t> UserConnCntMap_t;

msg_server

Er... msg_server is really too complex, let's leave it for later code analysis.

start-up

msg_server<----->login_server

msg_server will actively connect to login_server when it is started, and register its information with login_server through the data package CImPduMsgServInfo. Its processing logic is in LoginServConn.cpp:

1. void CLoginServConn::OnConfirm()  
2. {  
3.     log("connect to login server success\n");  
4.     m_bOpen = true;  
5.     g_login_server_list[m_serv_idx].reconnect_cnt = MIN_RECONNECT_CNT / 2;  
6.   
7.     uint32_t cur_conn_cnt = 0;  
8.     list<user_conn_t> user_conn_list;  
9.     CImUserManager::GetInstance()->GetUserConnCnt(&user_conn_list, cur_conn_cnt);  
10.     char hostname[256] = {0};  
11.     gethostname(hostname, 256);  
12.     CImPduMsgServInfo pdu(g_msg_server_ip_addr1.c_str(), g_msg_server_ip_addr2.c_str(),  
13.         g_msg_server_port, g_max_conn_cnt, cur_conn_cnt, hostname, MSG_SERVER_TYPE_TCP);  
14.     SendPdu(&pdu);  
15.   
16.     if (!user_conn_list.empty()) {  
17.         CImPduUserConnInfo pdu2(&user_conn_list);  
18.         SendPdu(&pdu2);  
19.     }  
20. }  

void CLoginServConn::OnConfirm()
{
    log("connect to login server success\n");
    m_bOpen = true;
    g_login_server_list[m_serv_idx].reconnect_cnt = MIN_RECONNECT_CNT / 2;

    uint32_t cur_conn_cnt = 0;
    list<user_conn_t> user_conn_list;
    CImUserManager::GetInstance()->GetUserConnCnt(&user_conn_list, cur_conn_cnt);
    char hostname[256] = {0};
    gethostname(hostname, 256);
    CImPduMsgServInfo pdu(g_msg_server_ip_addr1.c_str(), g_msg_server_ip_addr2.c_str(),
        g_msg_server_port, g_max_conn_cnt, cur_conn_cnt, hostname, MSG_SERVER_TYPE_TCP);
    SendPdu(&pdu);

    if (!user_conn_list.empty()) {
        CImPduUserConnInfo pdu2(&user_conn_list);
        SendPdu(&pdu2);
    }
}

When a user connects to msg_server and logs in successfully or the user disconnects, a CImPduUserCntUpdate packet is sent to login_server to notify login_server. Its processing logic is in LoginServConn.cpp:

1. void send_to_all_login_server(CImPdu* pPdu)  
2. {  
3.     CLoginServConn* pConn = NULL;  
4.   
5.     for (uint32_t i = 0; i < g_login_server_count; i++) {  
6.         pConn = (CLoginServConn*)g_login_server_list[i].serv_conn;  
7.         if (pConn && pConn->IsOpen()) {  
8.             pConn->SendPdu(pPdu);  
9.         }  
10.     }  
11. }  

void send_to_all_login_server(CImPdu* pPdu)
{
    CLoginServConn* pConn = NULL;

    for (uint32_t i = 0; i < g_login_server_count; i++) {
        pConn = (CLoginServConn*)g_login_server_list[i].serv_conn;
        if (pConn && pConn->IsOpen()) {
            pConn->SendPdu(pPdu);
        }
    }
}

msg_server<---->route_server

msg_server will actively connect route_server after starting, and report its current online user status to route_server through the data package CImPduOnlineUserInfo. Its processing logic is in LoginServConn.cpp:

1. void CRouteServConn::OnConfirm()  
2. {  
3.     log("connect to route server success\n");  
4.     m_bOpen = true;  
5.     m_connect_time = get_tick_count();  
6.     g_route_server_list[m_serv_idx].reconnect_cnt = MIN_RECONNECT_CNT / 2;  
7.   
8.     if (g_master_rs_conn == NULL) {  
9.         update_master_route_serv_conn();  
10.     }  
11.   
12.     list<user_conn_stat_t> online_user_list;  
13.     CImUserManager::GetInstance()->GetOnlineUserInfo(&online_user_list);  
14.     CImPduOnlineUserInfo pdu(&online_user_list);  
15.     SendPdu(&pdu);  
16. }  

void CRouteServConn::OnConfirm()
{
    log("connect to route server success\n");
    m_bOpen = true;
    m_connect_time = get_tick_count();
    g_route_server_list[m_serv_idx].reconnect_cnt = MIN_RECONNECT_CNT / 2;

    if (g_master_rs_conn == NULL) {
        update_master_route_serv_conn();
    }

    list<user_conn_stat_t> online_user_list;
    CImUserManager::GetInstance()->GetOnlineUserInfo(&online_user_list);
    CImPduOnlineUserInfo pdu(&online_user_list);
    SendPdu(&pdu);
}

When a user connects to msg_server and logs in successfully or the user disconnects, a CImPduUserStatusUpdate packet is sent to route_server to notify route_server. Its processing logic is in RouteServConn.cpp:

1. void send_to_all_route_server(CImPdu* pPdu)  
2. {  
3.     CRouteServConn* pConn = NULL;  
4.   
5.     for (uint32_t i = 0; i < g_route_server_count; i++) {  
6.         pConn = (CRouteServConn*)g_route_server_list[i].serv_conn;  
7.         if (pConn && pConn->IsOpen()) {  
8.             pConn->SendPdu(pPdu);  
9.         }  
10.     }  
11. }  

void send_to_all_route_server(CImPdu* pPdu)
{
    CRouteServConn* pConn = NULL;

    for (uint32_t i = 0; i < g_route_server_count; i++) {
        pConn = (CRouteServConn*)g_route_server_list[i].serv_conn;
        if (pConn && pConn->IsOpen()) {
            pConn->SendPdu(pPdu);
        }
    }
}

Call place

Calls that report local conditions to route_server and login_server are called in MsgConn.cpp when the user is online or offline:

1. void CMsgConn::SendUserStatusUpdate(uint32_t user_status)  
2. {  
3.   
4.     if (!m_bOpen) {  
5.         return;  
6.     }  
7.     CImUser* pImUser = CImUserManager::GetInstance()->GetImUserByName(m_user_name);  
8.     if (!pImUser) {  
9.         return;  
10.     }  
11.   
12.     //LoginServer is notified only by an offline notification  
13.     if (user_status == USER_STATUS_ONLINE) {  
14.         CImPduUserCntUpdate pdu(USER_CNT_INC, pImUser->GetUserId());  
15.         send_to_all_login_server(&pdu);  
16.           
17.         //if (pImUser->IsMsgConnEmpty()) {  
18.       
19.         CImPduUserStatusUpdate pdu2(USER_STATUS_ONLINE, pImUser->GetUserId(), GetClientTypeFlag());  
20.         send_to_all_route_server(&pdu2);  
21.         //}  
22.     } else if (user_status == USER_STATUS_OFFLINE) {  
23.         CImPduUserCntUpdate pdu(USER_CNT_DEC, pImUser->GetUserId());  
24.         send_to_all_login_server(&pdu);  
25.       
26.         //if (pImUser->IsMsgConnEmpty()) {  
27.         CImPduUserStatusUpdate pdu2(USER_STATUS_OFFLINE, pImUser->GetUserId(), GetClientTypeFlag());  
28.         send_to_all_route_server(&pdu2);  
29.         //}  
30.     }  
31. }  

void CMsgConn::SendUserStatusUpdate(uint32_t user_status)
{

    if (!m_bOpen) {
        return;
    }
    CImUser* pImUser = CImUserManager::GetInstance()->GetImUserByName(m_user_name);
    if (!pImUser) {
        return;
    }

    // LoginServer is notified only by an offline notification
    if (user_status == USER_STATUS_ONLINE) {
        CImPduUserCntUpdate pdu(USER_CNT_INC, pImUser->GetUserId());
        send_to_all_login_server(&pdu);
        
        //if (pImUser->IsMsgConnEmpty()) {
    
        CImPduUserStatusUpdate pdu2(USER_STATUS_ONLINE, pImUser->GetUserId(), GetClientTypeFlag());
        send_to_all_route_server(&pdu2);
        //}
    } else if (user_status == USER_STATUS_OFFLINE) {
        CImPduUserCntUpdate pdu(USER_CNT_DEC, pImUser->GetUserId());
        send_to_all_login_server(&pdu);
    
        //if (pImUser->IsMsgConnEmpty()) {
        CImPduUserStatusUpdate pdu2(USER_STATUS_OFFLINE, pImUser->GetUserId(), GetClientTypeFlag());
        send_to_all_route_server(&pdu2);
        //}
    }
}