1, Network communication and service
Network communication is the basic service of MySQL, including other related services derived from it, which constitutes the main way for MySQL client and server to complete interaction. The main functions include:
1. Network initialization and service initialization: including parameters, server and listener.
2. Network interaction module: data sending and receiving and control.
3. Interactive protocol module: including UNIX SOCKET protocol, TCP/IP protocol, pipeline and Share Memory protocol
These three parts basically cover the main contents of network communication and services.
2, Main process
The main processes are basically as follows:
The process of network communication is much simpler than that of Redis. The tasks related to distributed communication are ignored here for the time being.
3, Source code
Still go back to main - > mysqld in the previous article_ Main function:
bool my_init() {
char *str;
......
//Thread and parameter initialization
if (my_thread_global_init()) return true;
if (my_thread_init()) return true;
/* $HOME is needed early to parse configuration files located in ~/ */
if ((home_dir = getenv("HOME")) != nullptr)
home_dir = intern_filename(home_dir_buff, home_dir);
{
DBUG_TRACE;
DBUG_PROCESS(my_progname ? my_progname : "unknown");
#ifdef _WIN32
my_win_init();
#endif
MyFileInit();
......
return false;
}
}
static void my_win_init() {
DBUG_TRACE;
......
win_init_registry();
win32_init_tcp_ip();
MyWinfileInit();
}
static bool win32_init_tcp_ip() {
if (win32_have_tcpip()) {
WORD wVersionRequested = MAKEWORD(2, 2);
WSADATA wsaData;
/* Be a good citizen: maybe another lib has already initialised
sockets, so dont clobber them unless necessary */
if (WSAStartup(wVersionRequested, &wsaData)) {
/* Load failed, maybe because of previously loaded
incompatible version; try again */
WSACleanup();
if (!WSAStartup(wVersionRequested, &wsaData)) have_tcpip = 1;
} else {
if (wsaData.wVersion != wVersionRequested) {
/* Version is no good, try again */
WSACleanup();
if (!WSAStartup(wVersionRequested, &wsaData)) have_tcpip = 1;
} else
have_tcpip = 1;
}
}
return (0);
}
If you have written about the network development of Windows platform, the call of the last function is very clear. Now I see. Then down:
void notify_connect() {
#ifndef _WIN32
const char *sockstr = getenv("NOTIFY_SOCKET");
if (sockstr == nullptr) {
#ifdef WITH_SYSTEMD_DEBUG
sql_print_warning(
"NOTIFY_SOCKET not set in environment. sd_notify messages will not be "
"sent!");
#endif /* WITH_SYSTEMD_DEBUG */
return;
}
size_t sockstrlen = strlen(sockstr);
size_t sunpathlen = sizeof(sockaddr_un::sun_path) - 1;
if (sockstrlen > sunpathlen) {
std::cerr << "Error: NOTIFY_SOCKET too long" << std::endl;
LogErr(SYSTEM_LEVEL, ER_SYSTEMD_NOTIFY_PATH_TOO_LONG, sockstr, sockstrlen,
sunpathlen);
return;
}
//UDP communication initialization
NotifyGlobals::socket = socket(AF_UNIX, SOCK_DGRAM, 0);
sockaddr_un addr;
socklen_t addrlen;
memset(&addr, 0, sizeof(sockaddr_un));
addr.sun_family = AF_UNIX;
if (sockstr[0] != '@') {
strcpy(addr.sun_path, sockstr);
addrlen = offsetof(struct sockaddr_un, sun_path) + sockstrlen + 1;
} else { // Abstract namespace socket
addr.sun_path[0] = '\0';
strncpy(&addr.sun_path[1], sockstr + 1, strlen(sockstr) - 1);
addrlen = offsetof(struct sockaddr_un, sun_path) + sockstrlen;
}
int ret = -1;
do {
ret = connect(NotifyGlobals::socket,
reinterpret_cast<const sockaddr *>(&addr), addrlen);
} while (ret == -1 && errno == EINTR);
if (ret == -1) {
char errbuf[512];
LogErr(WARNING_LEVEL, ER_SYSTEMD_NOTIFY_CONNECT_FAILED, sockstr,
my_strerror(errbuf, sizeof(errbuf) - 1, errno));
NotifyGlobals::socket = -1;
}
#endif /* not defined _WIN32 */
}
Next, the initialization and configuration of a large number of relevant persistent parameters, and then the configuration of relevant functional interfaces controlled by macro definition. Next is authentication initialization:
void mysql_audit_initialize() {
#ifdef HAVE_PSI_INTERFACE
init_audit_psi_keys();
#endif
mysql_mutex_init(key_LOCK_audit_mask, &LOCK_audit_mask, MY_MUTEX_INIT_FAST);
memset(mysql_global_audit_mask, 0, sizeof(mysql_global_audit_mask));
}
bool Srv_session::module_init() {
if (srv_session_THRs_initialized) return false;
srv_session_THRs_initialized = true;
THR_stack_start_address = nullptr;
THR_srv_session_thread = nullptr;
server_session_list.init();
server_session_threads.init();
return false;
}
Next comes a batch of locks, semaphores and related initialization, including some custom string functions. Set and customize the stack space of the whole application thread.
static void set_ports() {
char *env;
if (!mysqld_port &&
!opt_disable_networking) { // Get port if not from commandline
mysqld_port = MYSQL_PORT;
/*
if builder specifically requested a default port, use that
(even if it coincides with our factory default).
only if they didn't do we check /etc/services (and, failing
on that, fall back to the factory default of 3306).
either default can be overridden by the environment variable
MYSQL_TCP_PORT, which in turn can be overridden with command
line options.
*/
#if MYSQL_PORT_DEFAULT == 0
struct servent *serv_ptr;
if ((serv_ptr = getservbyname("mysql", "tcp")))
mysqld_port = ntohs((u_short)serv_ptr->s_port); /* purecov: inspected */
#endif
if ((env = getenv("MYSQL_TCP_PORT")))
mysqld_port = (uint)atoi(env); /* purecov: inspected */
}
if (!mysqld_unix_port) {
#ifdef _WIN32
mysqld_unix_port = (char *)MYSQL_NAMEDPIPE;
#else
mysqld_unix_port = MYSQL_UNIX_ADDR;
#endif
if ((env = getenv("MYSQL_UNIX_PORT")))
mysqld_unix_port = env; /* purecov: inspected */
}
}
int delegates_init() {
alignas(Trans_delegate) static char place_trans_mem[sizeof(Trans_delegate)];
alignas(Binlog_storage_delegate) static char
place_storage_mem[sizeof(Binlog_storage_delegate)];
alignas(Server_state_delegate) static char
place_state_mem[sizeof(Server_state_delegate)];
alignas(Binlog_transmit_delegate) static char
place_transmit_mem[sizeof(Binlog_transmit_delegate)];
alignas(Binlog_relay_IO_delegate) static char
place_relay_io_mem[sizeof(Binlog_relay_IO_delegate)];
transaction_delegate = new (place_trans_mem) Trans_delegate;
if (!transaction_delegate->is_inited()) {
LogErr(ERROR_LEVEL, ER_RPL_TRX_DELEGATES_INIT_FAILED);
return 1;
}
binlog_storage_delegate = new (place_storage_mem) Binlog_storage_delegate;
if (!binlog_storage_delegate->is_inited()) {
LogErr(ERROR_LEVEL, ER_RPL_BINLOG_STORAGE_DELEGATES_INIT_FAILED);
return 1;
}
server_state_delegate = new (place_state_mem) Server_state_delegate;
binlog_transmit_delegate = new (place_transmit_mem) Binlog_transmit_delegate;
if (!binlog_transmit_delegate->is_inited()) {
LogErr(ERROR_LEVEL, ER_RPL_BINLOG_TRANSMIT_DELEGATES_INIT_FAILED);
return 1;
}
binlog_relay_io_delegate = new (place_relay_io_mem) Binlog_relay_IO_delegate;
if (!binlog_relay_io_delegate->is_inited()) {
LogErr(ERROR_LEVEL, ER_RPL_BINLOG_RELAY_DELEGATES_INIT_FAILED);
return 1;
}
return 0;
}
Continue to find network related parts, key cache and SSL encrypted communication:
bool process_key_caches(process_key_cache_t func) {
I_List_iterator<NAMED_ILINK> it(key_caches);
NAMED_ILINK *element;
while ((element = it++)) {
KEY_CACHE *key_cache = (KEY_CACHE *)element->data;
func(element->name, key_cache);
}
return false;
}
static void init_ssl() {
#if !defined(__sun)
#if defined(HAVE_PSI_MEMORY_INTERFACE)
static PSI_memory_info all_openssl_memory[] = {
{&key_memory_openssl, "openssl_malloc", 0, 0,
"All memory used by openSSL"}};
mysql_memory_register("mysqld_openssl", all_openssl_memory,
(int)array_elements(all_openssl_memory));
#endif /* defined(HAVE_PSI_MEMORY_INTERFACE) */
int ret = CRYPTO_set_mem_functions(my_openssl_malloc, my_openssl_realloc,
my_openssl_free);
if (ret == 0)
LogErr(WARNING_LEVEL, ER_SSL_MEMORY_INSTRUMENTATION_INIT_FAILED,
"CRYPTO_set_mem_functions");
#endif /* !defined(__sun) */
ssl_start();
}
void init_max_user_conn(void) {
hash_user_connections =
new collation_unordered_map<std::string, unique_ptr_my_free<user_conn>>(
system_charset_info, key_memory_user_conn);
}
Last but not least:
//Line 1740 defines the relevant listener and receiver
static Connection_acceptor<Mysqld_socket_listener> *mysqld_socket_acceptor =
nullptr;
#ifdef _WIN32
static Named_pipe_listener *named_pipe_listener = NULL;
Connection_acceptor<Named_pipe_listener> *named_pipe_acceptor = NULL;
Connection_acceptor<Shared_mem_listener> *shared_mem_acceptor = NULL;
#ifdef _WIN32
int win_main(int argc, char **argv)
#else
int mysqld_main(int argc, char **argv)
#endif
{
......
if (init_ssl_communication()) unireg_abort(MYSQLD_ABORT_EXIT);
if (network_init()) unireg_abort(MYSQLD_ABORT_EXIT);
......
#if defined(_WIN32)
if (mysqld_socket_acceptor != nullptr)
mysqld_socket_acceptor->check_and_spawn_admin_connection_handler_thread();
setup_conn_event_handler_threads();
#else
mysql_mutex_lock(&LOCK_socket_listener_active);
// Make it possible for the signal handler to kill the listener.
socket_listener_active = true;
mysql_mutex_unlock(&LOCK_socket_listener_active);
if (opt_daemonize) {
if (nstdout != nullptr) {
// Show the pid on stdout if deamonizing and connected to tty
fprintf(nstdout, "mysqld is running as pid %lu\n", current_pid);
fclose(nstdout);
nstdout = nullptr;
}
mysqld::runtime::signal_parent(pipe_write_fd, 1);
}
mysqld_socket_acceptor->check_and_spawn_admin_connection_handler_thread();
mysqld_socket_acceptor->connection_event_loop();
......
}
Look at network initialization:
static bool network_init(void) {
if (opt_initialize) return false;
#ifdef HAVE_SYS_UN_H
std::string const unix_sock_name(mysqld_unix_port ? mysqld_unix_port : "");
#else
std::string const unix_sock_name("");
#endif
std::list<Bind_address_info> bind_addresses_info;
if (!opt_disable_networking || unix_sock_name != "") {
if (my_bind_addr_str != nullptr &&
check_bind_address_has_valid_value(my_bind_addr_str,
&bind_addresses_info)) {
LogErr(ERROR_LEVEL, ER_INVALID_VALUE_OF_BIND_ADDRESSES, my_bind_addr_str);
return true;
}
Bind_address_info admin_address_info;
if (!opt_disable_networking) {
if (my_admin_bind_addr_str != nullptr &&
check_admin_address_has_valid_value(my_admin_bind_addr_str,
&admin_address_info)) {
LogErr(ERROR_LEVEL, ER_INVALID_ADMIN_ADDRESS, my_admin_bind_addr_str);
return true;
}
/*
Port 0 is interpreted by implementations of TCP protocol
as a hint to find a first free port value to use and bind to it.
On the other hand, the option mysqld_admin_port can be assigned
the value 0 if a user specified a value that is out of allowable
range of values. Therefore, to avoid a case when an operating
system binds admin interface to am arbitrary selected port value,
set it explicitly to the value MYSQL_ADMIN_PORT in case it has value 0.
*/
if (mysqld_admin_port == 0) mysqld_admin_port = MYSQL_ADMIN_PORT;
}
Mysqld_socket_listener *mysqld_socket_listener = new (std::nothrow)
Mysqld_socket_listener(bind_addresses_info, mysqld_port,
admin_address_info, mysqld_admin_port,
admin_address_info.address.empty()
? false
: listen_admin_interface_in_separate_thread,
back_log, mysqld_port_timeout, unix_sock_name);
if (mysqld_socket_listener == nullptr) return true;
mysqld_socket_acceptor = new (std::nothrow)
Connection_acceptor<Mysqld_socket_listener>(mysqld_socket_listener);
if (mysqld_socket_acceptor == nullptr) {
delete mysqld_socket_listener;
mysqld_socket_listener = nullptr;
return true;
}
if (mysqld_socket_acceptor->init_connection_acceptor())
return true; // mysqld_socket_acceptor would be freed in unireg_abort.
if (report_port == 0) report_port = mysqld_port;
if (!opt_disable_networking) assert(report_port != 0);
}
#ifdef _WIN32
// Create named pipe
if (opt_enable_named_pipe) {
std::string pipe_name = mysqld_unix_port ? mysqld_unix_port : "";
named_pipe_listener = new (std::nothrow) Named_pipe_listener(&pipe_name);
if (named_pipe_listener == NULL) return true;
named_pipe_acceptor = new (std::nothrow)
Connection_acceptor<Named_pipe_listener>(named_pipe_listener);
if (named_pipe_acceptor == NULL) {
delete named_pipe_listener;
named_pipe_listener = NULL;
return true;
}
if (named_pipe_acceptor->init_connection_acceptor())
return true; // named_pipe_acceptor would be freed in unireg_abort.
}
// Setup shared_memory acceptor
if (opt_enable_shared_memory) {
std::string shared_mem_base_name =
shared_memory_base_name ? shared_memory_base_name : "";
Shared_mem_listener *shared_mem_listener =
new (std::nothrow) Shared_mem_listener(&shared_mem_base_name);
if (shared_mem_listener == NULL) return true;
shared_mem_acceptor = new (std::nothrow)
Connection_acceptor<Shared_mem_listener>(shared_mem_listener);
if (shared_mem_acceptor == NULL) {
delete shared_mem_listener;
shared_mem_listener = NULL;
return true;
}
if (shared_mem_acceptor->init_connection_acceptor())
return true; // shared_mem_acceptor would be freed in unireg_abort.
}
#endif // _WIN32
return false;
}
It is divided into the creation of WIN and LINUX platforms. Just look at one of them. The focus is actually SQL / Conn_ For several files in the handler directory, templates are used here, so some jumps fail:
static inline bool spawn_admin_thread(MYSQL_SOCKET admin_socket,
const std::string &network_namespace) {
initialize_thread_context();
admin_thread_arg_t *arg_for_admin_socket_thread =
new (std::nothrow) admin_thread_arg_t(admin_socket, network_namespace);
if (arg_for_admin_socket_thread == nullptr) return true;
int ret = mysql_thread_create(
key_thread_handle_con_admin_sockets, &admin_socket_thread_id,
&admin_socket_thread_attrib, admin_socket_thread,
(void *)arg_for_admin_socket_thread);
(void)my_thread_attr_destroy(&admin_socket_thread_attrib);
if (ret) {
LogErr(ERROR_LEVEL, ER_CANT_CREATE_ADMIN_THREAD, errno);
return true;
}
wait_for_admin_thread_started();
return false;
}
bool Mysqld_socket_listener::check_and_spawn_admin_connection_handler_thread()
const {
if (m_use_separate_thread_for_admin) {
if (spawn_admin_thread(m_admin_interface_listen_socket,
m_admin_bind_address.network_namespace))
return true;
}
return false;
}
bool Mysqld_socket_listener::setup_listener() {
/*
It's matter to add a socket for admin connection listener firstly,
before listening sockets for other connection types be added.
It is done in order to check availability of new incoming connection
on admin interface with higher priority than on other interfaces..
*/
if (!m_admin_bind_address.address.empty()) {
TCP_socket tcp_socket(m_admin_bind_address.address,
m_admin_bind_address.network_namespace,
m_admin_tcp_port, m_backlog, m_port_timeout);
MYSQL_SOCKET mysql_socket = tcp_socket.get_listener_socket();
if (mysql_socket.fd == INVALID_SOCKET) return true;
m_admin_interface_listen_socket = mysql_socket;
if (!m_use_separate_thread_for_admin) {
m_socket_vector.emplace_back(mysql_socket, Socket_type::TCP_SOCKET,
&m_admin_bind_address.network_namespace,
Socket_interface_type::ADMIN_INTERFACE);
}
}
// Setup tcp socket listener
if (m_tcp_port) {
for (const auto &bind_address_info : m_bind_addresses) {
TCP_socket tcp_socket(bind_address_info.address,
bind_address_info.network_namespace, m_tcp_port,
m_backlog, m_port_timeout);
MYSQL_SOCKET mysql_socket = tcp_socket.get_listener_socket();
if (mysql_socket.fd == INVALID_SOCKET) return true;
m_socket_vector.emplace_back(mysql_socket, Socket_type::TCP_SOCKET,
&bind_address_info.network_namespace,
Socket_interface_type::DEFAULT_INTERFACE);
}
}
#if defined(HAVE_SYS_UN_H)
// Setup unix socket listener
if (m_unix_sockname != "") {
Unix_socket unix_socket(&m_unix_sockname, m_backlog);
MYSQL_SOCKET mysql_socket = unix_socket.get_listener_socket();
if (mysql_socket.fd == INVALID_SOCKET) return true;
Listen_socket s(mysql_socket, Socket_type::UNIX_SOCKET);
m_socket_vector.push_back(s);
m_unlink_sockname = true;
}
#endif /* HAVE_SYS_UN_H */
setup_connection_events(m_socket_vector);
return false;
}
const Listen_socket *Mysqld_socket_listener::get_listen_socket() const {
/*
In case admin interface was set up, then first check whether an admin socket
ready to accept a new connection. Doing this way provides higher priority
to admin interface over other listeners.
*/
#ifdef HAVE_POLL
uint start_index = 0;
if (!m_admin_bind_address.address.empty() &&
!m_use_separate_thread_for_admin) {
if (m_poll_info.m_fds[0].revents & POLLIN) {
return &m_socket_vector[0];
} else
start_index = 1;
}
for (uint i = start_index; i < m_socket_vector.size(); ++i) {
if (m_poll_info.m_fds[i].revents & POLLIN) {
return &m_socket_vector[i];
}
}
#else // HAVE_POLL
if (!m_admin_bind_address.address.empty() &&
!m_use_separate_thread_for_admin &&
FD_ISSET(mysql_socket_getfd(m_admin_interface_listen_socket),
&m_select_info.m_read_fds)) {
return &m_socket_vector[0];
}
for (const auto &socket_element : m_socket_vector) {
if (FD_ISSET(mysql_socket_getfd(socket_element.m_socket),
&m_select_info.m_read_fds)) {
return &socket_element;
}
}
#endif // HAVE_POLL
return nullptr;
;
}
Channel_info *Mysqld_socket_listener::listen_for_connection_event() {
#ifdef HAVE_POLL
int retval = poll(&m_poll_info.m_fds[0], m_socket_vector.size(), -1);
#else
m_select_info.m_read_fds = m_select_info.m_client_fds;
int retval = select((int)m_select_info.m_max_used_connection,
&m_select_info.m_read_fds, 0, 0, 0);
#endif
if (retval < 0 && socket_errno != SOCKET_EINTR) {
/*
select(2)/poll(2) failed on the listening port.
There is not much details to report about the client,
increment the server global status variable.
*/
++connection_errors_query_block;
if (!select_errors++ && !connection_events_loop_aborted())
LogErr(ERROR_LEVEL, ER_CONN_SOCKET_SELECT_FAILED, socket_errno);
}
if (retval < 0 || connection_events_loop_aborted()) return nullptr;
/* Is this a new connection request ? */
const Listen_socket *listen_socket = get_listen_socket();
/*
When poll/select returns control flow then at least one ready server socket
must exist. Check that get_ready_socket() returns a valid socket.
*/
assert(listen_socket != nullptr);
MYSQL_SOCKET connect_sock;
#ifdef HAVE_SETNS
/*
If a network namespace is specified for a listening socket then set this
network namespace as active before call to accept().
It is not clear from manuals whether a socket returned by a call to
accept() borrows a network namespace from a server socket used for
accepting a new connection. For that reason, assign a network namespace
explicitly before calling accept().
*/
std::string network_namespace_for_listening_socket;
if (listen_socket->m_socket_type == Socket_type::TCP_SOCKET) {
network_namespace_for_listening_socket =
(listen_socket->m_network_namespace != nullptr
? *listen_socket->m_network_namespace
: std::string(""));
if (!network_namespace_for_listening_socket.empty() &&
set_network_namespace(network_namespace_for_listening_socket))
return nullptr;
}
#endif
if (accept_connection(listen_socket->m_socket, &connect_sock)) {
#ifdef HAVE_SETNS
if (!network_namespace_for_listening_socket.empty())
(void)restore_original_network_namespace();
#endif
return nullptr;
}
#ifdef HAVE_SETNS
if (!network_namespace_for_listening_socket.empty() &&
restore_original_network_namespace())
return nullptr;
#endif
#ifdef HAVE_LIBWRAP
if ((listen_socket->m_socket_type == Socket_type::TCP_SOCKET) &&
check_connection_refused_by_tcp_wrapper(connect_sock)) {
return nullptr;
}
#endif // HAVE_LIBWRAP
Channel_info *channel_info = nullptr;
if (listen_socket->m_socket_type == Socket_type::UNIX_SOCKET)
channel_info = new (std::nothrow) Channel_info_local_socket(connect_sock);
else
channel_info = new (std::nothrow) Channel_info_tcpip_socket(
connect_sock, (listen_socket->m_socket_interface ==
Socket_interface_type::ADMIN_INTERFACE));
if (channel_info == nullptr) {
(void)mysql_socket_shutdown(connect_sock, SHUT_RDWR);
(void)mysql_socket_close(connect_sock);
connection_errors_internal++;
return nullptr;
}
#ifdef HAVE_SETNS
if (listen_socket->m_socket_type == Socket_type::TCP_SOCKET &&
!network_namespace_for_listening_socket.empty())
static_cast<Channel_info_tcpip_socket *>(channel_info)
->set_network_namespace(network_namespace_for_listening_socket);
#endif
return channel_info;
}
Without a good source code viewing tool, it is a little inconvenient to look at the code and wastes a lot of time. In fact, in the end, you will find that all the work has returned to the simplest network communication. Of course, some abstract classes are used for abstraction, and the four communication methods mentioned at first are formed into four classes to realize. This is also a good way, but compared with the previous Redis, the encapsulation is much simpler.
Under this folder, the communication is encapsulated as follows:
channel_info: the connection channel manager is actually the related management of the client
connection_acceptor: network communication receiver, which is responsible for receiving connection messages
connection_handler: connection handle, client handling mechanism
connection_handler_impl: interface of connection management mechanism
connection_handler_manager: Connection Manager
connection_handler_*_thread: thread processor required for connection
Before the source code, there is no secret.
4, Summary
The importance of network communication in MySql is not as important as that in Redis. After all, as a relational database, it can not bear the direct pressure of tens of millions. It is more the application of data storage and analysis. However, with the development of MySql, it is unknown whether it will go further on the distributed road or go another way in the future. After all, change is eternal and the future is unpredictable.
However, as an infrastructure, network communication can not be avoided in any case, which is the same in any scenario. Learning should be strong and weak, and distinguish between primary and secondary.
