I preface

Mycat exposes services in the form of a Server, and its main configuration class is Server XML, this article mainly focuses on some small details, mainly including:

• Server. How to load XML
• Configuration and role in Server
• Flow of Server in connection request

II Server.xml configuration

2.1 Server.xml configuration details

Server.xml is mainly loaded in XML schemaloader, server The XML file is mainly divided into the following parts

User management

<user name="user">
    <property name="password">user</property>
    // User accessible schemas can be separated by commas
    <property name="schemas">db001</property>
    <property name="readOnly">true</property>
    <property name="defaultSchema">db001</property>
</user>
Copy code

configuration management

The configuration file is pulled directly from the source code, which is also very clear. If you are interested, you can pull the source code to have a look

<system>
       <property name="nonePasswordLogin">0</property> <!-- 0 For login with password, 1 for login without password ,The default is 0. If it is set to 1, you need to specify the default account-->
       <property name="ignoreUnknownCommand">0</property><!-- 0 An unrealized message is encountered(Unknown command:),An error will be reported. 1 means to ignore the message and return ok Message.
In some mysql When the client has logged in, it will continue to send login messages,mycat Will report an error,This setting can bypass this error-->
       <property name="useHandshakeV10">1</property>
       <property name="removeGraveAccent">1</property>
       <property name="useSqlStat">0</property>  <!-- 1 To enable real-time statistics, 0 to disable -->
       <property name="useGlobleTableCheck">0</property>  <!-- 1 To enable all overtime consistency detection, and 0 to close -->
       <property name="sqlExecuteTimeout">300</property>  <!-- SQL Execution timeout unit:second-->
       <property name="sequenceHandlerType">1</property>
       <!--<property name="sequnceHandlerPattern">(?:(\s*next\s+value\s+for\s*MYCATSEQ_(\w+))(,|)|\s)*)+</property>
       INSERT INTO `travelrecord` (`id`,user_id) VALUES ('next value for MYCATSEQ_GLOBAL',"xxx");
       -->
       <!--Must have MYCATSEQ_perhaps mycatseq_Enter the sequence matching process. Note: MYCATSEQ_With spaces-->
       <property name="sequnceHandlerPattern">(?:(\s*next\s+value\s+for\s*MYCATSEQ_(\w+))(,|)|\s)*)+</property>
       <property name="subqueryRelationshipCheck">false</property> <!-- When there is an associated query in the sub query,Check whether there is a fragment field in the associated field .default false -->
       <property name="sequenceHanlderClass">io.mycat.route.sequence.handler.HttpIncrSequenceHandler</property>
       <!--  <property name="useCompression">1</property>--> <!--1 For on mysql Compression protocol-->
       <!--  <property name="fakeMySQLVersion">5.6.20</property>--> <!--Set simulated MySQL Version number-->
       <!-- <property name="processorBufferChunk">40960</property> -->
       <!--
       <property name="processors">1</property>
       <property name="processorExecutor">32</property>
        -->
       <!--Default to type 0: DirectByteBufferPool | type 1 ByteBufferArena | type 2 NettyBufferPool -->
       <property name="processorBufferPoolType">0</property>
       <!--The default is 65535 64 K be used for sql Maximum text length when parsing -->
       <!--<property name="maxStringLiteralLength">65535</property>-->
       <!--<property name="sequenceHandlerType">0</property>-->
       <!--<property name="backSocketNoDelay">1</property>-->
       <!--<property name="frontSocketNoDelay">1</property>-->
       <!--<property name="processorExecutor">16</property>-->
       <!--
           <property name="serverPort">8066</property> <property name="managerPort">9066</property>
           <property name="idleTimeout">300000</property> <property name="bindIp">0.0.0.0</property>
           <property name="dataNodeIdleCheckPeriod">300000</property> 5 * 60 * 1000L; //Connection idle check
           <property name="frontWriteQueueSize">4096</property> <property name="processors">32</property> -->
       <!--Distributed transaction switch: 0 means not filtering distributed transactions, 1 means filtering distributed transactions (if only global tables are involved in distributed transactions, then not filtering), and 2 means not filtering distributed transactions,However, distributed transaction logs are recorded-->
       <property name="handleDistributedTransactions">0</property>

       <!--
       off heap for merge/order/group/limit      1 On 0 off
   -->
       <property name="useOffHeapForMerge">0</property>

       <!--
           Unit is m
       -->
       <property name="memoryPageSize">64k</property>

       <!--
           Unit is k
       -->
       <property name="spillsFileBufferSize">1k</property>

       <property name="useStreamOutput">0</property>

       <!--
           Unit is m
       -->
       <property name="systemReserveMemorySize">384m</property>


       <!--Whether to adopt zookeeper Coordinated switching  -->
       <property name="useZKSwitch">false</property>

       <!-- XA Recovery Log Log path -->
       <!--<property name="XARecoveryLogBaseDir">./</property>-->

       <!-- XA Recovery Log Log name -->
       <!--<property name="XARecoveryLogBaseName">tmlog</property>-->
       <!--If yes true Strictly observe the isolation level if necessary,Not just select Statement to switch connections in a transaction-->
       <property name="strictTxIsolation">false</property>
       <!--If 0,Involving multiple DataNode of catlet Tasks do not execute across threads-->
       <property name="parallExecute">0</property>
   </system>
Copy code

Firewall configuration

<firewall>
   <whitehost>
      <host host="1*7.0.0.*" user="root"/>
   </whitehost>
   <blacklist check="false">
   </blacklist>
</firewall>
Copy code

2.2 Server.xml detail parsing

Server. There are several tags inside XML, mainly user, system and firewall

// User: used to define the user and permission of the logged in Mycat, which is finally mapped to UserConfig. In addition to the above, it also provides the following special attributes: 
- benchmark : Load balancing strategy , 0 The number of connections is unlimited
- privilegesConfig : Table level addition, deletion, modification and query settings


// System tag: system configuration tag, which is finally mapped to SystemConfig object. In addition to the above, it mainly has the following configurations
- charset : Configure character set , Be sure to be consistent with the database
- defaultSqlParser : Specifies the default parser
- processors : Number of threads available to the system (default Runtime.getRuntime().availableProcessors())
- processorBufferChunk : Each allocation Socket Direct Buffer , Here, the size of bytes obtained will be affected
- processorBufferPool : BufferPool Calculation proportion of
- processorBufferLocalPercent : ThreadLocalPool distribution Pool Scale size of , Default 100
- sequnceHandlerType : Mycat Type of global sequence

- private long processorCheckPeriod : clear NIOProcessor Idle interval
- private long dataNodeIdleCheckPeriod :  Back end connection cleanup interval
- private long dataNodeHeartbeatPeriod :  Initiate heartbeat interval for backend read / write
- private int useOffHeapForMerge : Enable Off Heap for Merge  1-Enabled, 0-Not enabled


- private int usingAIO = 0 : Open AIO 
- private int packetHeaderSize = 4 : MySQL Protocol message length
- private int maxPacketSize = 16 * 1024 * 1024 : Maximum size of data that can be carried
- private String memoryPageSize : Page size,corresponding MemoryBlock The size of the, in M
- private long idleTimeout : The timeout length of the idle time of the connection
- private int txIsolation : The isolation level of the initial front-end connection transaction , Corresponding 1-4

Copy code

III Server. How to load XML

Server.xml is loaded in the ConfigInitializer. Finally, the bit Map is loaded and passed to MycatServer

//Read server xml
XMLConfigLoader configLoader = new XMLConfigLoader(schemaLoader);

// Corresponding schema xml -> dataHost
private final Map<String, DataHostConfig> dataHosts;
// Corresponding schema xml -> dataNode
private final Map<String, DataNodeConfig> dataNodes;
// Corresponding schema XML rules
private final Map<String, SchemaConfig> schemas;
// Corresponding System global configuration
private final SystemConfig system;
// Corresponding User specific Config
private final Map<String, UserConfig> users;
// Corresponding firewall configuration
private final FirewallConfig firewall;
// Corresponding slice configuration
private final ClusterConfig cluster;
Copy code

IV Configured usage scenarios

4.1 node 1: Server connection

When connecting through the tool, the init DB operation will be performed first, and the DB information will be obtained to obtain the connection

C- FrontendConnection # initDB
public void initDB(byte[] data) {
   
   // S1: get data through MySQLMessage object
   // init request parameters: \ u0006 \ u0000 \ u0000 \ u0000 \ u0002db001 - > DB001

   // S2: check DB validity
   db == null || !privileges.schemaExists(db)
   
   // S3: verify whether the current user exists
   privileges.userExists(user, host)
 
   // S4: get Schemas
   Set<String> schemas = privileges.getUserSchemas(user);
   if (schemas == null || schemas.size() == 0 || schemas.contains(db)) {
      this.schema = db;
      // OkPacket.OK : 07 00 00 02 00 00 00 02 00 00 00
      write(writeToBuffer(OkPacket.OK, allocate()));
   } else {
      String s = "Access denied for user '" + user + "' to database '" + db + "'";
      writeErrMessage(ErrorCode.ER_DBACCESS_DENIED_ERROR, s);
   }
} 

Copy code

Supplement 1: okpocket OK: what does 700010002000 mean? MySQL OK package return structure

An OK packet is sent from the server to the client, indicating the successful completion of the command. In MySQL 5.7.5, OK packets are also used to indicate EOF, and the EOF packet has been discarded.

The packages will contain the following data: packet header, affected rows, last inserted ID, status flag status, etc. there will be an opportunity to find a case to see in detail later

Supplement 2: privileges object

As you can see, FrontendPrivileges frequently appears in the above code. What is the function of this object?

FrontendPrivileges is a permission provider interface. It provides several common methods, such as:

• schemaExists: check whether the schema exists
• userExists: check whether the user exists and can use host to implement isolation policy
• checkFirewallWhiteHostPolicy: check firewall policy

.... wait

It can be understood that FrontendPrivileges is for server The main implementation class of logical processing of XML data is MycatPrivileges

Its internal logic is also relatively simple, mainly dealing with MycatConfig, for example:

// Check whether the schema exists
public boolean schemaExists(String schema) {
    MycatConfig conf = MycatServer.getInstance().getConfig();
    return conf.getSchemas().containsKey(schema);
}


// Very good practice, directly return the instance object 
public static final MycatServer getInstance() {
    // private static final MycatServer INSTANCE = new MycatServer();
    // The static object has completed the initialization of related config in the constructor
    return INSTANCE;
}

Copy code

You can see that the internal configuration basically includes the common Config

4.2 node 2: configuration during query

The configuration will take effect in each phase. Here are some common scenarios:

Common usage scenarios of timeout

// Regularly check tasks and handle recycled resources: 
C- MycatServer # processorCheck : 
C- NIOProcessor # backendCheck
private void backendCheck() {
    // S1: get timeout
   long sqlTimeout = MycatServer.getInstance().getConfig().getSystem().getSqlExecuteTimeout() * 1000L;
   
   // S2: iterate over all connections
   Iterator<Entry<Long, BackendConnection>> it = backends.entrySet().iterator();

   // S3: first delete the empty connection, and then close the connection with SQL execution timeout
  if (c.isBorrowed() && c.getLastTime() < TimeUtil.currentTimeMillis() - sqlTimeout) {
      //.....
  }
     
}

// PS: there are many usage scenarios, so I won't look at them one by one here

Copy code

From this case, it is obvious that the configuration is mainly processed by obtaining the configuration from MycatServer, and MycatServer is one of the core objects of the whole process. Let's take a detailed look in the next article

V Data reception of Server

When Mycat NIO processes the request, you can see a Byte [] continuous flow. What data does this Byte contain?

// In FrontendConnection # initDB, we can see the specific data of data. Take a connection as an example: 
- \u0006\u0000\u0000\u0000\u0002 : prefix , Judge the specific type
- db001 : concrete Server Schema 

// Supplement: judge the C- FrontendCommandHandler # handle by the first four types
public void handle(byte[] data) {
    if (source.getLoadDataInfileHandler() != null && source.getLoadDataInfileHandler().isStartLoadData()) {
        MySQLMessage mm = new MySQLMessage(data);
        int packetLength = mm.readUB3();
        if (packetLength + 4 == data.length) {
            source.loadDataInfileData(data);
        }
        return;
    }
    
    // Can see
    switch (data[4]) {
        // public static final byte COM_INIT_DB = 2;
        case MySQLPacket.COM_INIT_DB:
            commands.doInitDB();
            source.initDB(data);
            break;
        // public static final byte COM_QUERY = 3;    
        case MySQLPacket.COM_QUERY:
            commands.doQuery();
            source.query(data);
            break;
        case MySQLPacket.COM_PING:
            commands.doPing();
            source.ping();
            break;
        case MySQLPacket.COM_QUIT:
            commands.doQuit();
            source.close("quit cmd");
            break;
        //.....  Omit part type
        default:
            commands.doOther();
            MycatConfig config = MycatServer.getInstance().getConfig();
            if (config.getSystem().getIgnoreUnknownCommand() == 1) {
                LOGGER.warn("Unknown command:{}", data[4]);
                source.ping();
            } else {
                LOGGER.error("Unknown command:{}", new String(data));
                source.writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR,
                        "Unknown command");
            }
    }
}
Copy code

Generally, a query is divided into two parts:

• Initialize connection
• Execute the current operation, taking Query as an example

It can be seen here that Byte[0] - Byte[4] is still a flag bit, and specific SQL will be passed in later Whether initDB or Query, a key step will be taken:

MySQLMessage mm = new MySQLMessage(data);
// Parsing flag bit
mm.position(5);
String db = mm.readString();


// When the label bit parsing is completed, the corresponding Handler will be called to complete the subsequent logic
- queryHandler.query(sql);

Copy code

summary

After a simple study of the entrance, reference to the documents and some supplements, you can slowly go deep into the whole system in the follow-up

last