Another city! Two communities work together to create a new tool for API log monitoring

Posted by gernot on Thu, 16 Dec 2021 12:30:12 +0100

Since Apache RocketMQ entered the vision of global developers in 2016, it has developed into the core data base of technology in e-commerce, finance, education, science and technology and other fields.

According to incomplete statistics, more than 70% of domestic users (including top 100 enterprises in various fields such as finance, insurance, wealth and securities companies) have deployed Apache RocketMQ on a large scale on the core application link, including the world's top five cloud manufacturers, which have also launched cloud products and services related to Apache RocketMQ.

In addition to the routine application to the processing of core business messages, many companies have begun to use Apache rocketmq for log processing and analysis.

Plug in introduction

In order to meet the needs of enterprise users for log processing, Apache APIs IX has released a log plug-in RocketMQ logger based on Apache RocketMQ, which supports pushing API interface request logs to the RocketMQ cluster in JSON.

The plug-in uses the TCP protocol natively supported by RocketMQ and realizes the functional features of high concurrency and high-performance access through the non blocking TCP Socket API provided by OpenResty.

At the same time, the API log format sent by using the rocketmq logger plug-in is the same as that of other log plug-ins. It also supports batch sending logs, custom log format, retry and other functions.

In addition, the plug-in also supports TLS encrypted transmission and configuring AK and SK authentication to access Apache RocketMQ to meet users' needs for data security.

How to use

Start RocketMQ

First, start RocketMQ locally with the following commands. Please refer to the specific steps Official documents.

wget https://dlcdn.apache.org/rocketmq/4.9.2/rocketmq-all-4.9.2-bin-release.zip
unzip rocketmq-all-4.9.2-bin-release.zip
cd rocketmq-4.9.2/
nohup sh bin/mqnamesrv &
nohup sh bin/mqbroker -n 127.0.0.1:9876 -c conf/broker.conf &

Open the plug-in in Apache APIs IX

In a production environment, you can enable the rocketmq logger plug-in for a specified route by executing only one command.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
   "plugins": {
      "rocketmq-logger": {
          "nameserver_list" : [ "127.0.0.1:9876" ],
          "topic" : "test",
      }
   },
   "upstream": {
      "nodes": {
          "127.0.0.1:1980": 1
      },
      "type": "roundrobin"
   },
   "uri": "/hello"
}'

After the rocketmq logger plug-in is enabled, any request for endpoint URI/hello will push the log to Apache RocketMQ.

Refer to the following table for details of specific supported parameters:

Plug in metadata settings

Of course, if you do not want to use the default log format during use, you can also set metadata for the plug-in.

First, you can adjust the format of related logs in the form of templates.

After the log format adjustment is completed, you need to report to / apifix / admin / plugin_ The metadata endpoint sends a request to update metadata. For details, refer to the code below.

curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/rocketmq-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "log_format": {
        "host": "$host",
        "@timestamp": "$time_iso8601",
        "client_ip": "$remote_addr"
    }
}'

Disable plug-ins

If you no longer use the plug-in, you can disable the rocketmq logger plug-in by deleting the corresponding JSON configuration in the plug-in configuration. This process does not need to restart the service. Enter the code below to take effect immediately.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
   "methods": ["GET"],
   "uri": "/hello",
   "plugins": {},
   "upstream": {
       "type": "roundrobin",
       "nodes": {
           "127.0.0.1:1980": 1
       }
   }
}'

Want to learn more about practice?

On Sunday, December 26, Apache apisik community will work with Apache RocketMQ community to bring you more practice content to share online. If you want to learn more application practice content, join us in this online Meetup! Many wonderful topics are waiting for you to see.

Group communication

Scan the QR code below and join the Apache apisik online live communication group to learn more about community dynamics!

Topics: RocketMQ gateway Cloud Native apisix

Programmer Think