AR human-computer interaction
Augmented Reality (AR) has ushered in a new age of human-computer interaction. Currently we’re quite comfortable using a keyboard and mouse to interact with our computers (which was not always the case) or tapping and swiping our mobile devices. However, augmented reality changes the game! Out goes the mouse and in comes gestures, voice recognition, and eye tracking!
Types of AR User Interactions
User interaction is a key component of AR development and it’s crucial to use the right combination of AR user interactions to provide a comfortable and efficient user experience.
Controllers are devices that track position and rotation, register button presses, and can be represented as digital hands to the user – most commonly used in Virtual Reality (VR) experiences. This enables users to interact with digital objects by touching, holding or even throwing within a virtual environment. Less common within AR experiences, hand controllers can be used in the form of a small mouse-sized hand clicker.
Hand tracking provides a more natural interaction method than using physical controllers. AR devices such as the HoloLens can detect hand position and interpret them as instructions, allowing interaction between the user’s hands and the digital objects within the AR experience.
Speech recognition is widely used with systems such as Siri, Alexa and Google Assistant commonplace and uses spoken words to launch menu items and trigger events. This method of interaction provides less friction than typing input on a virtual keyboard.
For screen-based AR (mobile phones and tablets), touchscreen interaction can occur in the form of a tap, swipe or pinch. Depending on the type of touch, the user can then select or affect the parameters of a digital object such as position, rotation or scale.
Controllers, gestures, voice, and touch are all ‘explicit inputs' where the user is directly providing an instruction. Implicit inputs are indirect interactions that still communicate with the virtual environment and include head or camera movement, and eye tracking.
Device tracking can be achieved via the user’s head position for digital eyewear or mobile phone or tablet in screen-based AR. These devices come equipped with cameras and sensors that detect position, depth, tilt, and acceleration in real time. Two key sensors – the accelerometer and gyroscope – calculate changes in movement and position. The accelerometer provides accurate data on acceleration, velocity, direction, and speed of movement. The gyroscope provides data on 3D movement such as orientation and rotation.
Eye tracking software – used in head-mounted displays – is used to detect the position of the eye and use it as input instructions to point at and select items. A benefit of using eye-tracking is that you can have a mixture of high and low resolution displays based on the user’s current focus to reduce processing requirements.
The implicit interactions of movement and eye tracking can be combined with the explicit interactions of touch, gestures, and controllers to provide a satisfying and immersive control system to your AR experience.
AR User Interaction Tips
- Don’t rely on a single user interaction input type. Have multiple ways of accessing a menu.
- Use gestures sparingly. Input that requires frequent hand gestures can make the user fatigued.
- Mix both implicit and explicit interaction types, such as gaze to hover with a device’s position and a gesture to select.
- Don’t use eye tracking as the primary input method as it can be uncomfortable for the user.
- Allow direct manipulation of digital objects in 3D space rather than via 2D onscreen buttons.
- Prototype and test often to validate assumptions and try alternative user input types.
- Use interaction types that your target audience will be most comfortable with.