It's time to adapt perfect input support for all kinds of devices

Posted by SoireeExtreme on Thu, 27 Jan 2022 04:46:55 +0100

With the progress of technology and the popularity of cross platform applications, your Android applications are no longer limited to running on straight board touch-screen devices. Richer interaction enables users to use your application with more complex input. So as a developer, it's time to consider providing strong input support for a variety of devices. This article has prepared for you to share more extensive and powerful input support. You are welcome to read it.

If you prefer to learn about this through video, please check here:

https://www.bilibili.com/vide...

△ it's time to adapt perfect input support for all kinds of equipment

For various Android devices, Input It is one of the key elements that determine the user application experience. Developers usually want user interaction to be as simple and intuitive as possible, but if you buy a new foldable device with a keyboard, and your favorite application does not support standard key operation, this experience will be very frustrating.

At present, many Android devices have built-in non touch input, such as Chromebook keyboard, and even some devices provide it as standard. So how should developers ensure that different input methods are suitable for their applications in various Android device types, such as mobile phones, foldable devices, tablets, chromebooks, chromeboxes with external display, Chromebase with built-in display, Android TV and so on.

△ multiple Android devices

You should know that not all users use the mobile touch screen to interact with your application. Some users may use keyboards and stylus, and even some users have Accessibility Demand. So every developer needs to spend some time thinking about how to make the application bring a pleasant experience to as many users as possible?

Enhanced input

△ standard input mode and enhanced input mode

For each type of input device, we can divide the application functions into standard use cases and enhanced use cases:

Standard use cases include the functions expected by users, such as selection, text input, long press and right click. The technical implementation of processing such use cases is relatively simple and can be executed automatically in some cases. If you want to provide more unique and enhanced support for your application, you need to think more about it.
In the enhanced use case, some functions are not only better, but may be necessary. For example, a mobile game that does not support game consoles and a text editor that does not support standard copy and paste shortcuts are not welcomed by users.

In addition to providing basic functions, we should consider adding unique functions that can support users, which is the way for developers to truly stand out from their applications. For example, the application eDJing shown below adds support for keyboard disc playing, touch pad disc rubbing and MIDI DJ controller, and the user's mobile phone or Chromebook can immediately become a DJ workstation.

△ eDJing application scenario

Cubasis is a first-class digital audio workstation application. It has released a new version based on Chrome OS operating system optimization. With the advantages of large screen and the convenience of MIDI controller connecting to Chromebook, it not only enhances the function, but also improves the practicability of the application.

△ Cubasis application scenario

Drawing applications pay more attention to the continuous normal operation of Bluetooth and USB drawing board, and the application of low delay stylus API to drawing and drawing applications in Chrome OS operating system.

△ application scenarios of drawing

In short, users want to get a unique, pleasant and intuitive experience in your application, and this experience is built by you!

Keyboard input support

The keyboard is built into the Chromebook or becomes part of the user's daily experience using removable devices, tablets, foldable devices and TVs. The good news is that most basic keyboard input can usually be used directly, unless you're working on building your own on-screen keyboard or writing your own text input view from scratch.

Send key support (KEYCODE_ENTER)

Developers need to create a new line for the Enter key in the standard edit text view. If your application has chat function, feedback form, briefing registration or any function that needs to send text, the default line feed behavior is certainly not what you expect. Don't worry, the sending function you expect is easy to achieve.

△ chat sending text

The developer needs to create a new line for the Enter key in the standard EditText view. The code for pressing the Enter key is displayed here. The complete code is as follows:

override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {
    when (keyCode) {
        KEYCODE_ENTER -> {     // Undo onKeyUp and listen for KEYCODE_ENTER
            sendMessage()
            return true     // If the application has handled the key event, make sure to return true
        }
    }
    // If the event is not handled, return it to the system for processing
    return super.onKeyUp(keyCode, event)  // If not, pass the event to super
}```

**Media key support (Media key)**

If your application wants to support media playback, you also need to include media key support. To do this, please Enter In key code KEYCODE_ENTER Replace it with the media key code you want to support. For example, it is used here MEDIA_NEXT and MEDIA_PREV. In addition, please do not forget to use SPACE To use the spacebar to control playback and pause. The complete code is shown below:

override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {

when (keyCode) {
    KEYCODE_SPACE, KEYCODE_MEDIA_PLAY_PAUSE -> {
          playOrpauseMedia(); 
          return true 
    }
    KEYCODE_MEDIA_NEXT -> {nextTrack(); return true }
    KEYCODE_MEDIA_PREV -> {prevTrack(); return true }
}
// If the event is not handled, return it to the system for processing
return super.onKeyUp(keyCode, event)

}

**Game key support (KEYCODE_W|A|S|D)**

For game applications, you may want to include pairs of arrow keys and W,A,S,D Four button support, which is also very simple. You just need to Android Found the correct in the document [Key code](https://developer.android.google.cn/reference/android/view/KeyEvent) and listen to these keys. The complete code is as follows:

when (keyCode) {

KEYCODE_W, DPAD_UP -> { goUp(); return true }
KEYCODE_A, DPAD_LEFT -> { goLeft(); return true }
KEYCODE_S, DPAD_DOWN -> { goDown(); return true }
KEYCODE_D, DPAD_RIGHT -> { goRight(); return true }

}

It should be noted that it is usually necessary to monitor when providing keyboard support onKeyUp，This way, you don't have to worry about sending duplicate messages when you press and hold a key onKeyDown event. In addition, if you want to ensure millisecond time response, you can listen onKeyDown And handle repeated key events by itself. The complete code is shown below:

override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {

when (keyCode) {
    KEYCODE_F -> {
        shootFireball()
        return true
    }
}
return super.onKeyUp(keyCode, event)

}

For more information, please visit: [ChromeOS.dev](http://chromeos.dev)

**Shortcut key support (Ctrl+)**

In addition to the basic keyboard keys, the configuration based on Ctrl Shortcut keys, such as copy, paste, undo, redo, etc. common shortcut keys are suitable for many applications. Excellent keyboard support will help your application achieve more functions. Some applications even further put advanced functions within the reach of users, such as users in use eDJing To apply, simply press and hold Ctrl Key, you can use the touch pad to rub dishes.

![](https://devrel.andfun.cn/devrel/posts/2022/01/ro5gyI.png)

The following shows the for undo Ctrl+Z Shortcut key code, which is similar to the previous code onKeyUp and onKeyDown Code, but used dispatchKeyShortcutEvent To indicate the meta key combination. Note that the button is pressed at this time Alt,Ctrl or Shift Key to trigger this operation. The complete code is shown below:

override fun dispatchKeyShortcutEvent(event: KeyEvent): Boolean {

return when (event.getKeyCode()) {
    KEYCODE_Z -> {
        undoAction() // Triggered by [Ctrl]+z, [Shift]+z, or [Alt]+z
        true
    }
    else -> {
        super.dispatchKeyShortcutEvent(event)
    }
}

}

If you only want to respond to specific meta key combinations, you can use Meta_Alt_On，Similar can also be used isShiftPressed() or isCtrlPressed() Methods. The complete code is shown below:

when {

event.isCtrlPressed && event.isShiftPressed -> {
    redoAction(); true // [Ctrl]+[Shift]+z trigger
}
event.isCtrlPressed -> {
    undoAction(); true //[Ctrl]+z trigger
}
else -> { // Leave it to the system for processing
    super.dispatchKeyShortcutEvent(event)
}

}

According to the code here, you can only press it at the same time Ctrl+Z The undo operation will only be performed when the is pressed, and also only when the is pressed at the same time Ctrl+Shift+Z The redo operation will be performed only when the Alt. 

## **Mouse / touchpad input support**

Like keyboards, most mouse and touchpad inputs usually work without any additional code. But developers still need to use the mouse to test all the functions of the application to see if there are any omissions.

![△ mouse](https://devrel.andfun.cn/devrel/posts/2022/01/NBfbw4.png)

△ mouse

**Right click support**

Right click is one of the most common omissions. A common example of touch is a long press on the screen to perform context based operations, but long press mouse clicks are not intuitive. If you want to support right-click, there are several options here. The code for adding context menu is shown here. The complete code is shown below:

registerForContextMenu(myView)
//First register one or more views for the context menu, which will automatically handle long press and right-click operations.
...
override fun onCreateContextMenu(menu, view, menuInfo) {

super.onCreateContextMenu(menu, view, menuInfo)
menuInflater.inflater(R.menu.myContextMenu, menu)
// Then undo the overrideonCreateContextMenu and expand the correct menu layout. The same menu can be used for multiple views.

}
override fun onCreateContextMenu(item: MenuItem): Boolean {

return when(item.itemId){
    R.id.myMenuItemId -> { menuItemAction(); true }
    else -> { return super.onContextItemSelected(item) }
    // Finally, setting onContextItemSelected indicates what needs to be done when a particular menu item is selected.
}

}

For other right-click behaviors outside the context menu, you can use onContextClickListener To set up the view, simply use it to call the same method used in the long click case. The corresponding codes are as follows::

myView.setOnContextClickListener {

   performContextAction()
   true

}

**Hover response support**

![](https://devrel.andfun.cn/devrel/posts/2022/01/0U0eEG.png)

When users use the mouse or touchpad, they usually want the interface to respond in some way. For example, when the mouse cursor hovers over the clickable view, visual feedback will be generated. As shown in the figure, the pointer icon may have changed, or some other visual instructions appear, which can be intuitively felt by the user. The code when the pointer hovers over the view is shown here. The complete code is shown below:

myView. Setonhoverlistener {view, motionevent - > / / set onHoverListener for the view

when (motionEvent.action) {
    ACTION_HOVER_ENTER -> {
        view.pointerIcon = PointerIcon.getSystemIcon(
            appIicationContext, PointerIcon.TYPE_HAND
        )
        view.setBackgroundColor(Color.BLUE)
        true
    } // Monitor HOVER_ENTER event and perform corresponding actions. According to the code, the pointer icon will change to hand shape and the background color will change to blue.
    ACTION_HOVER_EXIT -> { resetUI(); }
    else -> { false }
    // Don't forget to set when hover_ Reset icon and background colors when exit occurs.
}

Pointer capture support

Pointer capture is another common mouse and touchpad enhancement that is not only critical for some games, but also adds specific features to some applications. The application can capture the mouse cursor through the pointer capture function so that the cursor does not appear on the screen, so that the relative pointer events can be received without moving the cursor to the edge of the screen. First person perspective games such as Minecraft: education are good examples.

△ Minecraft: Education Edition

To support pointer capture, you can first call requestPointerCapture, and then call releasePointerCapture to release the captured pointer. OnCapturedPointerListener can be added to the code to use the received pointer data, and use the relative change of pointer position to realize some great functions. The complete code is as follows:

view.requestPointerCapture()
...
view.releasePointerCapture()
...
view.setOnCapturedPointerListener { _, motionEvent ->
    // Calculates the coordinate increment since the last motionEvent event
    mouseChangeX = motionEvent.x
    mouseChangey = motionEvent.y

    // Calculates the displacement since the last motionEvent event
    val totalDistance = hypot(mouseChangeX, mouseChangeY)
    doSomethingCool(totalDistance)
}

Gesture support

In addition, the Android API also provides advanced pointer processing, which can add gesture, double finger opening and zooming support, as shown in the figure below.

If you want to learn more about the Android API, please refer to the Android Developer website for a getting started guide—— Use touch gestures.

Stylus input support

If you've added excellent support for pointers to your application, the stylus usually works as expected for most use cases. Some enhancements of the stylus are very noteworthy. For example, some devices support stylus tilting and pressing, which helps you add some excellent controls and functions in painting or drawing applications. In addition, there is a low latency stylus API that allows you to get the lowest latency display response in painting or drawing applications, and provides configurable stroke prediction to create a pen drawing experience on paper. For practical results, check out Concepts like apps on supported chromebooks or Android devices.

△ Concepts app supports stylus

if need Understand implementation details , please refer to the Android Developer website for axis_ Press and axis_ Document for tilt.
Low latency stylus API library and demo applications, see GitHub.

Using the stylus in the Android simulator

We work with Microsoft to introduce the host stylus support into the Android simulator. If you are optimizing the application to provide more advanced stylus support, you will be able to use the Android simulator to test accurate tilt and press control on the supported host.

The Microsoft Surface Duo developer experience team worked with Google to support advanced input development and testing, such as multi touch analysis and stylus support. So how to test the application with the stylus when running the application in the simulator?

△ test the stylus in the simulator

As shown in the figure, the Microsoft Duo 2 simulator running on the Surface Studio is displayed. At the moment, there are two applications running at the same time: the right pane is a sample application that allows you to test the pressing sensitivity, pen direction, erase pen tip and other pen buttons of the stylus; The left pane is the Microsoft OneNote application. Using the simulator, you can draw, take notes or erase on the OneNote canvas.

We are very excited. On touch enabled PC s, the Android simulator can now support multi touch, which allows you to test gestures that require multiple fingers to interact with applications, such as double finger opening, zooming and other touch interactions.

△ use gestures in Google Maps

This collapsible simulator with built-in Android Studio is running Google maps. You can zoom in and out of the map with just two fingers. Moreover, these simulators are not only updated to support the use of only two fingers, but can support up to 10 touch points if your hardware allows.

All of these changes you see are not unique to the Surface Duo simulator, but they also apply to other collapsible simulators. Microsoft has also been using these simulator updates to develop and optimize its own applications, such as testing the interaction of the stylus on a variety of device types, including Surface Duo, large screen and other collapsible devices.

Handle input support

If you have a game app, you need to add joystick support. Use the appropriate key code to determine whether you want to perform an operation on onKeyUp or onKeyDown.

△ game console

The key codes of the joystick direction key and the keyboard arrow key are the same. Just listen for KEYCODE_DPAD events can handle both. The joystick buttons also have their own key codes. You can monitor these buttons, just like the setting for the X button here. The complete code is as follows:

when (keyCode) {
    BUTTON_X -> {
        engageNitro(); return true
    }
    DPAD_UP, KEYCODE_W -> {
        goForward(); return true
    }
    DPAD_LEFT, KEYCODE_A -> {
        turnLeft(); return true
    }
    ...
}

MIDI input support

When special hardware is involved, support needs to be provided for devices and use cases. MIDI support is particularly important for music and creative tools. It allows a wide range of expressive input signals, from pressure-sensitive piano keyboards to devices with many different input triggers such as sliders, knobs, keyboards, etc.

If you want to use or learn more about MIDI, there are some convenient open source examples in the Android Media Samples library to help you get started with midi. Please check MidiSynth Project, especially the Synth Engine module in the project. Note that starting with Android 29, you can also use NDK MIDI API.

review

Large screen Android devices have emerged and become more and more popular. Providing excellent input support on Android has always been important, especially for foldable devices, tablets and Chrome operating system. Please think about the input processing of your application and how to increase interaction, unlock new functions and improve the application experience. We look forward to all developers investing in building wonderful Android applications and adding excellent input processing to them!

Welcome click here Submit feedback to us, or share your favorite content and found problems. Your feedback is very important to us. Thank you for your support!

Topics: Android

Programmer Think