Telepresence robot interface

— Excuse me, Mr. Wozniak?
— Oh, hey. Nice virtual presence device.
— Thank you. I just wanna say I'm a big fan. One of my proudest possessions is a vintage 1977 Apple II. Despite the file system limitations of Apple DOS 3.3... it was a pretty nifty little achievement.
— Thanks, we were shooting for nifty.
— You know, if you had it here, I'd autograph it for you.
— Don't move for 15 to 30 minutes... depending on how the busses are running.
— Nerds.
(Big Bang Theory, S04EP02)

***

The problem of communication has long been solved by the Internet, and now it is time to deal with the remote interaction.

The telepresence robot is a screen on wheels that travels around the office and transmits the image and voice of, for example, the CEO, who cannot attend the general meeting.

Nowadays this is quite an extravagant thing. The production of robots far from the mass, the price ranges from thousands to several tens of thousands of dollars. Devices may look different: you can meet all kinds of shapes from a clumsy peace of iron to an elegant plastic stick with huge screen on a very mobile basis.

Telepresence robots: Giraffe, RP-7i, VGO, TiLR, Texai, Anybots QA, Double Robotics, Anybots QB, Sparky Jr.

However, this research is not so much about the appearance of robots, which is the task for industrial designers, but about the interface which allows to manage these vehicles.

For author, it was a shock to discover that the interface of these toys of several thousand dollars each is such an ugly thing. Interfaces of robots Gostai Jazz and Botiful:

Therefore, when one Russian company decided to make a good interface for its robot, it was an excellent opportunity to distinguish the product against the others.

Control analysis

The construction of the robot is such that it allows it to move in any directions: left, right, forward, back, diagonally across two diagonals, and also raise and lower the head.

Not all movements are equally useful. There are basic moves, without which the movement would be incomplete, and through which the remaining actions can be realized. There are 4 such moves:

And the remaining moves are following:

ИTheir implementation can be reduced to a combination of basic motion operations. There are also two special actions that are worth mentioning in a separate category:

Obviously, without 4 basic actions, robot control is inadequate, so when designing the interface, they should be given the most attention. Without two head actions, you can control the robot, but it's not at all sweet, because very often the object of observation is above or below the level of eyes. These actions should also be given a special attention.

As for the three additional control commands, they do not improve anything: practice shows that the sideways movements or strafes are convenient only in Quake 3.

Basic control

The robot is controlled from the browser of a desktop computer or laptop. This means that we have a keyboard and one of the manipulator devices: a mouse or a laptop touchpad.

Immediately, the first obvious association is revealed: the main actions 1-4 are uniquely placed on keyboard shortcuts-arrows. Perhaps, this association is so good that it will allow even a beginner to guess from the first time how to control a robot.

Head control is a bit more complicated. In general, raising and lowering the head is also well associated with the up and down keys ↑, but they are already occupied, and there are no easy-to-use keys on the keyboard nearby.

Following the keyboard, the view falls on the mouse. The idea to use a mouse for machinations with a head is not bad, moreover, the mouse can rotate the camera in all directions, and not just raise and lower it.

But let’s not hurry. If you just assign the head control to the mouse, then you need to remove the rotation from the arrows to the left ← and right →, because otherwise the mouse movement may contradict the pressing of the buttons. When controlling the mouse, the buttons should not rotate the robot, but move it to the left and right. However, we agreed that these actions are not basic, and they should not be paid attention when the main operations are not yet implemented.

In general, control from the keyboard and the mouse is possible and successfully practiced in computer games, not with arrows though, but with the WASD keys. The arrows are too close to the right edge of the keyboard, so I would have to crawl over the table to avoid pulling my left arm.

Then the mouse wheel comes to mind. What if you control the lift by scrolling up and down? This is also not the best idea: the scrolling speed can be very different in different systems, and some models of the mouse (Apple Magic Mouse) are too sensitive and often scroll accidentally.

In fact, the only more or less acceptable solution for the mouse is the introduction of a screen slider that can be lifted up and down by holding down the left mouse button, similar to scrolling the scroll bar.

But this approach also has many drawbacks. User will often have to shift his hand from the keyboard to the mouse to adjust the incline. This significantly reduces the speed of interaction with the interface. So we'll do this: let's leave the slider control as the most obvious way for beginners, and then we'll introduce an advanced control method from the keyboard.

Our goal is to control the robot with one hand. Therefore, we will not connect the second hand and allocate two more control keys. We will use what was rejected from the very beginning — the busy buttons up ↑ and down ↓ and add the modifier key Ctrl.

When the Ctrl is pressed, the up key ↑ and down ↓ will control the tilt of the head, and without pressing Ctrl they will still control the movement of the robot. That is simple: one-handed operation, fastest interaction, closest to the arrow keys on all popular keyboards (Cmd and Control for small and large Apple Keyboards). True, you can not both go forward and move your head, but it's even good.

So, we placed 6 basic actions on 4 keys and entered one modifier key. Now it remains to deal with additional operations.

Additional control

As already mentioned above, usually side movements are placed on the A and D keys. Thus, the entire set of additional actions is successfully placed on the WASD, where W is the forward motion, S is the backward motion, A is the left movement, D is the movement to the right. When using this set, you can also move along the 4 corners: to the right and forward, to the right and back, to the left and back, to the left and forward.

With this approach, it is necessary to fully control the camera, and not just raising and lowering the head. Such management is most reasonable to provide with the mouse, which we rejected when considering standard actions. To manage advanced actions, the use of the mouse only has benefits: in the first stage, the task was to make the interface as simple and obvious as possible, now the task is to create advanced control.

The WASD keys and the mouse are typical controls in 3D computer games. With only one difference: in the game, control works instantly, without having almost no delay between the user's action and recall in the game. In the case of a robot, the control is performed via the Internet, and the command time alone can reach several seconds with poor communication. The second factor is robot mechanics, which has its own delay. The robot does not react to the forward command instantly, because you need to activate the wheels and leave the resting state.

In delay, as such, there is nothing to worry about, it will not disable the robot, but the lack of activity on the screen can confuse the user: is the system alive? For this reason, you should enter feedback, including mouse control.

In addition, it is worthwhile to enter any indicators of robot status. For example, the level of the signal Wi-Fi, battery power and time before the end of the control session.

It is also necessary to show the control on the screen and provide a replacement for the keyboard via mouse control, to make sure that the user can start safely right away without reading the guide.

And here our review goes to the actual design.

Design of the control

To show the main interface association, the connection of the 4 main actions with the arrow keys, we show the keys on the screen, in the form of a triangle, in the lower right corner.

It is necessary to pay attention to the design of keys. The controls should be clearly visible on any background, because the background in life can be a white wall, or someone's black jacket. At the same time, it is necessary to make them not distracting. Therefore, we use the old technique and make the inner part of the elements white, and we paint the borders black and add a small white shadow to the arrows.

Adding a rotation of the head with the Ctrl or Cmd key.

It does not look obvious. It is clear that the user will press Ctrl while acquainting themselves with the interface, but he needs to be shown that when something is clicked something has changed and that the key needs to be kept in control. To do this, we enter the feedback and show that when the Ctrl key is pressed, the up arrow ↑ and down ↓ go into the camera control mode.

Unfortunately, in the world there is no understandable camera control icon that could be placed on the keys. If you use the eye icon, you get a face. The wavy arrows seem too strange. After long torment we stop on the simplest symbol: a point.

Now, when you click on Ctrl, the user will receive a sign in the form of dots and a highlighted for a second forward key ↑, and this should be perceived as a call for simultaneous pressing Ctrl and ↑, which will cause the camera to rise. So there is an independent familiarization with the interface.

But the new users will not be forgotten: they will get a camera tilt adjuster that will move in conjunction with Ctrl + ↑ ↓.

By the way, look at these head control icons made especially for this project.

The basic control is finished. As for WASD control, it will not be on the screen at all, otherwise there will be a heap of buttons. For feedback, all the same basic buttons can be highlighted.

Feedback for the mouse can be represented as a wheel (or joystick), which occurs when the mouse is pressed and disappears when released.

A mouse in the form of a circular pin cursor freely moves inside the wheel, but has no way to leave it. The closer the cursor to the borders, the faster the camera rotates. Moving up, right, down and left causes the head to rise, turn the robot to the right, lower the head and turn the robot to the left. In this case, a full 360 degree view is possible. This wheel is essentially an alternative to an analog circular joystick with game controllers.

Well, and the turnaround, obviously, should be hung on the spacebar.

Other elements design

The robot's interface, of course, is not limited to control. There are also additional elements, such as: audio control, pause, help, screen size and indicators. The most important indicators are as follows:

The standard place for indicators is the top center of the screen. Since there are enough space on the screen, we will use the icons with captions.

All listed indicators are event-based: they should be shown only if necessary. The signal level icon is the icon of a weak signal, the battery level is only needed when the battery is running out, the time until the end of the session is the same.

The appearance of the icon should be accompanied by an animation of three blinks or similar. In special cases, you can use color coding: for example, at 10% of the battery charge, display the icon in yellow.

From auxiliary actions, the following comes to mind:

To place the pause buttons and mute the microphone, the left upper corner fits perfectly.

By the way, the old problem: should the button for turning off the microphone be simple or crossed out? Strikethrough means, on the one hand, the status: the microphone is turned off. On the other hand, this means action: turn off the microphone.

Bottom left, we can put video zoom and help buttons, and on the right, there is a duplicating camera showing the user’s face.

Total summary

So, it's time to end up with little things and see the whole picture. That's what we’ve got.

A few words about a competitior

After this interface done, I learned about the development of Double Robotics. This elegant robot favorably differs from the others in appearance:

And in aspect of control interface too.

The interface attracts with its unbelievable simplicity. It seems that its authors have managed to throw out all the additional functions.

Actually this is not true. The development of Double Robotics is currently managed only from the iPad, which means that the mute and video mute buttons, pause and volume control are on the device body, so they do not need to be included in the interface.

The camera is also simpler: all control is placed on the screen, and the robot is controlled simultaneously with two hands, so the questions about the speed of interaction and the choice of suitable input devices are simply not relevant.

An excellent, albeit obvious solution is the unbreakable control keys, which allow you to simply move your hand across the screen and do not tear off your fingers when you need to change the direction of motion.

Unfortunately, all these techniques are not available from a simple desktop computer.