IoTDB It is the first open source time series database project initiated by Tsinghua University and is now the top project of Apache. IoTDB can provide users with services such as data collection, storage and analysis. Due to its lightweight architecture, high performance and high availability, and seamless integration with Hadoop and Spark ecology, it meets the needs of massive data storage, high-throughput data writing and complex data query and analysis in the field of industrial IoT.
EMQ x is a large-scale, elastic and scalable open source cloud native distributed Internet of things message middleware, which is released by EMQ Yingyun technology, an open source Internet of things data infrastructure software supplier. EMQ x can efficiently and reliably handle the concurrent connection of massive Internet of things devices, and has built-in powerful rule engine function to conduct high-performance real-time processing of event and message flow data. The rule engine provides a flexible "configuration" business integration scheme through SQL statements, simplifies the business development process, improves ease of use, and reduces the coupling between users' business logic and EMQ X.
This article will introduce how to use the MQTT data bridging function of EMQ X rule engine to receive the data sent by MQTT client and insert it into the timing database IoTDB in real time.
preparation
The software and environment used in this example:
- Operating system: Mac OSX
- IoTDB: Binary package (Server), version 0.12.4
- MQTT server: EMQ X open source v4 three point one one
- MQTT client software: MQTTX v1.6.0
IoTDB installation
First, we need to start from IoTDB official page Download the binary package of IoTDB Server (stand-alone version).
After downloading, unzip and enter the unzipped Directory:
% ls LICENSE README.md RELEASE_NOTES.md data ext licenses sbin NOTICE README_ZH.md conf docs lib logs tools
To enable the MQTT protocol support of IoTDB, you need to change the configuration file conf / IoTDB engine properties:
*Subsequent modeling uses a storage group root.sg , in order to increase the write parallelism, iotdb engine Virtual in properties_ storage_ group_ Num is set to the number of machine cores.
#################### ### MQTT Broker Configuration #################### # whether to enable the mqtt service. enable_mqtt_service=true # the mqtt service binding host. mqtt_host=0.0.0.0 # the mqtt service binding port. mqtt_port=2883 # the handler pool size for handing the mqtt messages. mqtt_handler_pool_size=1 # the mqtt message payload formatter. mqtt_payload_formatter=json # max length of mqtt message in byte mqtt_max_message_size=1048576
Where enable_mqtt_service defaults to false and needs to be changed to true. mqtt_ The default value of port is 1883. In order to avoid conflict with the port number of emqx, it needs to be changed to 2883.
Then use/ sbin/start-server.sh start IoTDB server:
% ./sbin/start-server.sh --------------------- Starting IoTDB --------------------- Maximum memory allocation pool = 2048MB, initial memory allocation pool = 512MB If you want to change this configuration, please check conf/iotdb-env.sh(Unix or OS X, if you use Windows, check conf/iotdb-env.bat). 2022-01-10 14:15:31,914 [main] INFO o.a.i.d.c.IoTDBDescriptor:121 - Start to read config file file:./sbin/../conf/iotdb-engine.properties ... 2022-01-10 14:14:28,690 [main] INFO o.a.i.d.s.UpgradeSevice:73 - Upgrade service stopped 2022-01-10 14:14:28,690 [main] INFO o.a.i.db.service.IoTDB:153 - Congratulation, IoTDB is set up successfully. Now, enjoy yourself! 2022-01-10 14:14:28,690 [main] INFO o.a.i.db.service.IoTDB:101 - IoTDB has started
Let's keep this terminal window still and open a new command line terminal window to start IoTDB's shell tool:
% ./sbin/start-cli.sh --------------------- Starting IoTDB Cli --------------------- _____ _________ ______ ______ |_ _| | _ _ ||_ _ `.|_ _ \ | | .--.|_/ | | \_| | | `. \ | |_) | | | / .'`\ \ | | | | | | | __'. _| |_| \__. | _| |_ _| |_.' /_| |__) | |_____|'.__.' |_____| |______.'|_______/ version 0.12.4 IoTDB> login successfully IoTDB>
At this point, the IoTDB environment is ready. If you want to know the basic usage of IoTDB, you can refer to the on the official website Quick start page.
Install and configure EMQ X
Download and launch EMQ X
We directly use the command line to download the macOS version of EMQ X open source version. For more installation packages, please visit EMQ X open source download page.
% wget https://www.emqx.com/en/downloads/broker/4.3.11/emqx-macos-4.3.11-amd64.zip
Then unzip and start EMQ X:
% unzip -q emqx-macos-4.3.11-amd64.zip % cd emqx % ./bin/emqx console log.to = "console" Erlang/OTP 23 [erts-11.1.8] [emqx] [64-bit] [smp:8:8] [ds:8:8:8] [async-threads:4] [hipe] Starting emqx on node emqx@127.0.0.1 Start mqtt:tcp:internal listener on 127.0.0.1:11883 successfully. Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully. Start mqtt:ws:external listener on 0.0.0.0:8083 successfully. Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully. Start mqtt:wss:external listener on 0.0.0.0:8084 successfully. Start http:management listener on 8081 successfully. Start http:dashboard listener on 18083 successfully. EMQ X Broker 4.3.11 is running now! Eshell V11.1.8 (abort with ^G) (emqx@127.0.0.1)1>
Configuration rules
Open with browser EMQ X Dashboard , create a rule on the rule engine page:
The SQL statement is:
SELECT
clientid,
now_timestamp('millisecond') as now_ts_ms,
payload.bar as bar
FROM
"t/#"
Then we add a "bridge data to MQTT Broker" action to the rule at the bottom of the page:
This action needs to be associated with a resource. We click "new resource" in the upper right corner to create an MQTT Bridge resource:
For the remote Broker address, fill in the MQTT service address of IoTDB, i.e. "127.0.0.1:2883". The client Id, user name and password are all filled in root, because root is the default user name and password of IoTDB.
Other options remain the same as the default value. Click the "test connection" button to ensure that the configuration is correct, and then click the "new" button in the lower right corner to create resources.
Now return to the action creation page, and the drop-down box of associated resources is automatically filled with the resources we just created.
Now let's continue to fill in more action parameters:
IoTDB doesn't care about the message subject. We fill in an arbitrary subject: foo.
IoTDB requires that the message content is in JSON format, and the message content template can be filled in according to the style in the figure above. For details, see IoTDB's Communication service agreement document.
{
"device": "root.sg.${clientid}",
"timestamp": ${now_ts_ms},
"measurements": [
"bar"
],
"values": [
${bar}
]
}
Pay attention to the“ c l i e n t i d " , " {clientid}", " ClientID ", {now_ts_ms}" and "${bar}" are variables extracted from the output of the regular SQL statement, so these variables must correspond to the SELECT sentence of the SQL statement.
Now you can click "confirm" to save the action configuration, and then click "new" again to complete the creation of the rule.
Sending messages using MQTT Client
Next, we use the MQTT client tool - MQTT X to send a message to EMQ X:
MQTT X is a fully open source MQTT 5.0 cross platform desktop client released by EMQ. It supports the rapid creation of multiple MQTT client connections online at the same time, which is convenient to test the connection, publishing and subscription functions of MQTT/TCP, MQTT/TLS, MQTT/WebSocket and other MQTT protocol features.
In the connection parameters of MQTT client, we only need to fill in one parameter, Client ID: "abc", and the others remain unchanged by default.
After the connection is successful, we send two messages with the subject of "t/1", and the message content format is:
{
"bar": 0.2
}
Then go back to the rule engine page of EMQ X Dashboard, observe the number of hits of the rule, and confirm that the rule has been triggered twice:
Finally, we return to the IoTDB client window of the command line terminal and use the following SQL statement to query the data:
IoTDB> SHOW TIMESERIES root.sg.abc +---------------+-----+-------------+--------+--------+-----------+----+----------+ | timeseries|alias|storage group|dataType|encoding|compression|tags|attributes| +---------------+-----+-------------+--------+--------+-----------+----+----------+ |root.sg.abc.bar| null| root.sg| FLOAT| GORILLA| SNAPPY|null| null| +---------------+-----+-------------+--------+--------+-----------+----+----------+ Total line number = 1 It costs 0.006s IoTDB> SELECT * FROM root.sg.abc +-----------------------------+---------------+ | Time|root.sg.abc.bar| +-----------------------------+---------------+ |2022-01-10T17:39:41.724+08:00| 0.3| |2022-01-10T17:40:32.805+08:00| 0.2| +-----------------------------+---------------+ Total line number = 2 It costs 0.007s IoTDB>
Data insertion succeeded!
epilogue
So far, we have completed the persistence of messages to IoTDB timing database through EMQ X rule engine function.
In the actual production scenario, we can use EMQ X to process a large number of concurrent connections of IOT devices, flexibly process business functions through the rule engine, and then persist the messages sent by the devices to IoTDB database. Finally, we can use Hadoop/Spark, Flink or Grafana to connect IoTDB to realize big data analysis and visual display.
The combination of EMQ X + IoTDB is a simple, efficient, scalable and highly available server-side integration scheme, which is a good choice for the scenario of Internet of things device management and data processing.
Copyright notice: This article is EMQ original, please indicate the source for reprint.
Original link: https://www.emqx.com/zh/blog/store-mqtt-messages-to-time-series-database-iotdb