The original text was published by natural growth in the TesterHome community, click Original link Can communicate directly with the author.
1, Foreword
When using the ios devices of WeTest and Testin remotely, you will always feel that the operation is not smooth, because the mobile phone will Move only when you release the mouse. It should be because XCUITest is used in WDA to realize Tap operation, and its one-time operation needs a complete action chain, which cannot be decomposed into Down, Move and Up operations as in Android.
Recently found a software called wormhole, address: https://er.run/ , it can realize smooth remote operation. After research, it is found that its principle is to send Bluetooth mouse operation signal to the mobile phone through Bluetooth, so as to realize a pointer device to control the device.
At present, only one DEMO has been implemented, and its practicability needs to be improved.
2, Basic knowledge
1. Bluetooth HID device
The Human Interface Device (HID) defines the protocol, characteristics and application procedures of Bluetooth in man-machine interface devices. Typical applications include Bluetooth mouse, Bluetooth keyboard, Bluetooth game console and so on. The protocol is adapted from USB HID Protocol.
HID establishes two channels: Control Channel and Interrupt Channel. Control Channel is mainly used to transmit control packets. Packets transmitted in this channel are called synchronous reports. Packets transmitted in Interrupt Channel do not need confirmation, so they are called asynchronous reports.
The packets sent by different devices are different. The keyboard packets and mouse packets will be listed below.
2. Bluetooth SDP protocol
The SDP protocol allows the Client application to discover the services provided by the existing server application and the properties of these services. SDP only provides the mechanism of discovering services, not the method of using these services. Each Bluetooth device needs an SDP Service, except for Bluetooth devices that only serve as clients.
3. Bluetooth mouse signal
The biggest difference from using XCUITest is that the xy position of the Bluetooth mouse is not the position of the screen, but the relative position of the current point of the mouse, which is unfavorable for the implementation of Tap operation, because the relative position needs to be converted into the absolute position in the screen.
4. Auxiliary control in IOS
Different from the Bluetooth keyboard, supporting the Bluetooth mouse function needs to be turned on actively. The position is in Settings > auxiliary functions > touch > auxiliary functions. When the Bluetooth mouse signal is successfully sent, a small dot will appear on the mobile phone, and the Bluetooth device will also appear in Settings > auxiliary functions > touch > auxiliary functions > device.
3, Concrete implementation
1. Basic source code
Address: https://github.com/bonyadmitr...
This project supports Swift5, but only realizes the function of simulating the Bluetooth keyboard. As long as you run, connect the mobile phone to your Mac through Bluetooth, and then you can read the keyboard input of your Mac and transfer it to the mobile phone. PS: there seems to be something wrong with Monterey's Bluetooth connection. ios devices can't connect to Bluetooth at all.
2. Modify SDP and add mouse Report structure
There is an hidreportdescriptor in the project Txt file, you can see the Report structure of the keyboard, which needs to be added to the sdp protocol, but it is not an sdp protocol file. The real sdp file is serialportdictionary plist.
0206 - hiddescriptorlist > item 0 > item 1 > dataelementvalue is hidreportdescriptor Txt format, add the Report structure of the mouse into it.
3. Define mouse Report
Let's first look at the defined keyboard report, which needs to be combined with hidreportdescriptor Let's see.
let bytes: [UInt8] = [
0xA1, // 0 DATA | INPUT (HIDP Bluetooth)
0x01, // 1 Report ID
modifier, // 2 Modifier Keys
0x00, // 3 Reserved
keyCode, // 4 Keys ( 6 keys can be held at the same time )
0x00, // 5
0x00, // 6
0x00, // 7
0x00, // 8
0x00, // 9
0x00 // 10 Reserved
]The first byte A1 represents the HID device, and the second byte Report ID 01 represents the keyboard, which can correspond to 0x85, 0x01, // Report ID (1)
Next, modifier represents the function key, which can mean pressing multiple function keys at the same time. For example, 0b1 represents pressing the control key, 0b10 represents pressing the shift key, and 0b11 represents pressing the control and shift keys at the same time.
Although the reserved field is marked on the comment, the third byte below is actually meaningful, but it is only useful for output. It represents some locks, such as case lock and keypad lock.
0x05, 0x08, // Usage Page (LEDs) 0x19, 0x01, // Usage Minimum (Num Lock) 0x29, 0x05, // Usage Maximum (Kana) 0x95, 0x05, // Report Count (5) 0x75, 0x01, // Report Size (1) 0x91, 0x02, // Output (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile) 0x95, 0x01, // Report Count (1) 0x75, 0x03, // Report Size (3) 0x91, 0x01, // Output (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
Bytes 4 to 9 represent keys. You can press six keys at the same time. Only one key is supported in this project.
0x05, 0x07, // Usage Page (Kbrd/Keypad) 0x19, 0x00, // Usage Minimum (0x00) 0x2A, 0xFF, 0x00, // Usage Maximum (0xFF) 0x95, 0x06, // Report Count (6) 0x75, 0x08, // Report Size (8)
The 10th byte is a reserved field.
0x05, 0xFF, // Usage Page (Reserved 0xFF) 0x09, 0x03, // Usage (0x03) 0x75, 0x08, // Report Size (8) 0x95, 0x01, // Report Count (1)
Thus, the Report structure of the mouse found on the Internet can be written into the package format in the project.
0x05, 0x01, // Usage Page (Generic Desktop Ctrls) 0x09, 0x02, // Usage (Mouse) 0xA1, 0x01, // Collection (Application) 0x85, 0x02, // Report ID (2) 0x09, 0x01, // Usage (Pointer) 0xA1, 0x00, // Collection (Physical) 0x05, 0x09, // Usage Page (Button) 0x19, 0x01, // Usage Minimum (0x01) 0x29, 0x03, // Usage Maximum (0x03) 0x15, 0x00, // Logical Minimum (0) 0x25, 0x01, // Logical Maximum (1) 0x75, 0x01, // Report Size (1) 0x95, 0x03, // Report Count (3) 0x81, 0x02, // Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position) 0x75, 0x05, // Report Size (5) 0x95, 0x01, // Report Count (1) 0x81, 0x01, // Input (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position) 0x05, 0x01, // Usage Page (Generic Desktop Ctrls) 0x09, 0x30, // Usage (X) 0x09, 0x31, // Usage (Y) 0x09, 0x38, // Usage (Wheel) 0x15, 0x81, // Logical Minimum (-127) 0x25, 0x7F, // Logical Maximum (127) 0x75, 0x08, // Report Size (8) 0x95, 0x03, // Report Count (3) 0x81, 0x06, // Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position) 0xC0, // End Collection 0xC0, // End Collection
It can be seen that the Report ID becomes 2, the next byte (3+1*5 bit) represents the mouse button, and the last three bytes represent the moving distance of X, Y and Wheel respectively, which are also signed.
Therefore, the format of the mouse can be defined as follows:
let bytes: [Int8] = [
// 0b10100001, // 0 DATA | INPUT (HIDP Bluetooth)
-0b01011111,
0x02, // 0 Report ID
Action, // 0 just move 1 left_button 2 middle_button 3 right_button
Rel_X, // x
Rel_Y, // y
0x00 // scroll
]In order to express the negative number of xy conveniently, it is defined as Int8 format, and the first byte is changed from 0xA1 of UInt8 to Int8 format.
Since the original sendBytes method only supports the [UInt8] parameter, it needs to be changed to the generic method sendBytes (Channel: iobluetooth l2capchannel, bytes: [t])
4. Operation method
First define a sendMouse method
func sendMouse(Action: Int8, Rel_X: Int8, Rel_Y: Int8){
sendData(bytes: hidMouseReport(Action: Action, Rel_X: Rel_X, Rel_Y: Rel_Y))
}When the Action is 0, the cursor is moved, and when the Action is 1, it is pressed. In the process of pressing and holding, the Action of all operations is 1. When sending a package with Action 0, it is regarded as a release operation.
If you want to experience it quickly, you can send the keyboard trigger binding to this method, or you can cycle through a section of operation in the program.
4, Summary
At present, this idea has only been preliminarily realized, and there is still a lot of work to be done in the follow-up, such as:
1. Implement the external access interface, such as socket service or rpc to make the external call control methods.
2. Device management, the control between multiple Bluetooth devices cannot conflict, and the specified Bluetooth device can be identified at the same time.
3. It is convenient to encapsulate the pointer in the process of Up and Down into the relative position of the atom in the screen according to the operation of Up and Down.
The above is today's sharing. Have you learned it~
Want to learn more about dry goods and cutting-edge technology?
Want to get to know the test industry and elite?
Welcome to the 2022 MTSC Conference (the 10th China Internet test and Development Conference) ↓↓