A Sketch-based Smartphone Interface for Robotic Arm Control
Category: #User Interface Design & Evaluation, #Human Robot Interaction, #Sensor, #Touch Based Interface, #Mobile App #Android UI #Robotics
Adviser : Dr. Sonia Chernova
Date: Fall 2016 


RoboCollab has transformed the smartphone in to a robot controller. It is a sketch-based user interface to control KUKA robots. Designed hand gestures in interaction with smartphone and utilizing embedded sensors into robot operation techniques, provide many advantages in controlling the robot in initial steps of task manipulation by non-expert users.  This work-in-progress study challenges some features of existing teach pendants and propose solutions for 1. Effective multi-axis positioning the robot end-effector through a sketch-based user interface, 2. Teleoperation and remote control of the robot, 3. Distributed control by multiple operators in task manipulations and their turn-taking in sub-tasks manipulations. By conducting usability tests in the future, this research will evaluate the effectiveness of the proposed system from the users’ perspective.


Teach pendants are the most common UIs for controlling and programming industrial robots. They are very well suited for many features such as navigating through setups and control menus using the touch-based interface, manipulating the robot end-effector using the joystick, etc. However, they still have usability issues when doing a variety of tasks specifically when there is no prewritten code for manipulation tasks and operators are in the initial tests and exploration processes. In these situations, operators need to intuitively explore task completion workflows, avoid potential obstacles in robot workspace, draw different trajectories for robot arm maneuvering etc. To be more specific, there are cases in which the interaction techniques with robot is not effectively implemented in teach pendants. For instance:

  • Pointwise and continuous movements: industrial settings require the robot to move to a given object, either directly or via a given path. There must be a simple way of specifying target points for the robot to maneuver to and paths to follow while avoiding obstacles. The existing method seems cumbersome to non-expert users.

  • Multi-axis positioning the Robot End-effector (RE) with single movements

  • RE orientation to any specific pose by single tapping

  • Varying the scale of movement steps

  • Teleoperation and remote control of the robot

  • Distributed control by multiple operators in task manipulations and their turn-taking in sub-tasks manipulations.

To deal the above-mentioned challenges, this work-in-progress research project proposes a sketch-based smartphone application as an effective UI to control industrial robots. It evaluates the following hypotheses and questions. 

Hypotheses & Questions

  • H1: Using a smartphone application, the users will demonstrate a higher efficiency and lower error rate than when using the teaching pendant.

  • H2: Sketch-based interface is a more usable interface in comparison with existing teach pendants in terms of pointwise and continuous movements and multi-axes positioning the RE with single movements.

  • H3: Smartphone is a more usable interface in comparison with existing teach pendants in terms of RE orientation.

  • Q1: Can a smartphone provide the opportunity for teleoperation and remote control of the robot arm?

  • Q2: Can a smartphone be a robot controller interface for multiple operators in their task manipulations and turn-takings in sub-tasks completions.

To validate the hypotheses and answer the questions this study has conducted a pilot usability test and will do a user study and usability test with real users in future steps to analyze the effectiveness of the proposed interface from users’ perspective.

Sketch-based Interface

Based on usability challenges in current teach pendants, an alternative approach is sketch-based interfaces, which are rapidly gaining popularity with the increased uptake of consumer-level touch sensitive devices. Such interfaces have been devised for robot navigation, controlling multiple robots and household robots. The premise is that a user draws a free form picture on the screen and these sketches are then interpreted by software, providing robot commands for task manipulations, navigation, etc. Sketch-input is a promising approach because of its intuitive pen-and-paper metaphor and is available on a wide variety of platforms such as PCs, iPhones, and interactive white boards. Furthermore, it has been shown that sketching encourages creativity and enables users to concentrate on the overall problems rather than details. Although there are many potentials in the integration of sketch-based devices into robot controlling process, no specific interface with such feature has been implemented for industrial robots’ manipulations. This research implements such a feature in robot controller’s UI.

Smartphone as a UI to Control Industrial Robots

Nowadays, smartphones are one of the most ubiquitous micro electro mechanical systems that are equipped with various sensors such as gyroscope, accelerometer-based tilt sensors, touch interface, cameras and many others. All these options make smartphones a popular, simple and effective controller for many tools and applications including the control of robots. So, this research explores how such smartphones can be used as robot controller and let single or multiple operators control the robot either locally or in teleoperation mode.

Technical Implementation

System Components

There are three main components in the proposed system. First, a smartphone application as a responsive UI for controlling the robotic arm. Second, A virtual robot and its configuration for real-time task manipulation.  Inserting such virtual robot into workflow letting us perform tasks safely while we are still in development process. And third, a 6-Axis KUKA industrial robot. Table 1 explains all involved hardware and software components in more detail.

System Overview:

Figure 1 demonstrates a general overview of the system from sketch-based UI to the operation of the robot. The following steps are done in the implementation process.

  • Network configuration: The establishment of wireless connection via User Datagram Protocol(UDP) to stream data to Grasshopper environment and Transmission Control Protocol/Internet Protocol (TCP/IP) in communication with KRC4 robot controller. 

  • Utilizing embedded sensors in smartphone like capacitive touch, gyroscope and accelerometer to orient and translate RE.

  • Designing a sketch-based interface that provides functionalities like relative and absolute positioning for RE.

  • The configuration of a virtual robot in the Grasshopper environment with some basic movement routines

  • The potential integration of a vision system to map the captured view of work space on user-interface for both accurate task manipulation and the opportunity of teleportation of robot arm.

general overview of the system from sketch-based UI to the operation of the robot
Developed tabs in smartphone application: 2) network setup and IP address assignment. 3) Relative positioning for translation and orientation of the robot end-effector. 4) Traversing from point A to point B. 5) Movement through a continuous path. 6) Utilities tab

User Interface Functionalities and Human Gestures to Control RE

Relative positioning to specific target points includes translation in x, y and z axes and orientation around x and y axes. There are key benefits in fulfilling such functionality. First, in contrast to common teach pendants in which for each rotation degree, operator needs to rotate the joystick or click a button, orientation can be accomplished with just two tapping for setting the first and last poses. The second benefit is providing a relative translation of the end-effector with variable scales. Setting such adjustable scales for movement steps, allow for user-defined movement granularity 

Sketch-based interface allows positioning the robot end-effector both point wise and in continuous trajectories. In contrast with common teach pendants, the sketch-based interface lets RE movements between origin and destination points just by tapping on the screen for each desired position. In continuous trajectory drawing, sliding on screen can create the path for robot arm movement. 

Virtual Robot Control

A virtual robot is configured in Grasshopper environment with some basic movement routines based on Inverse and Forward Kinematics calculations (IK Solver and FK solver components). To manipulate such virtual robot, streamed data from smartphone is receiving by UDP listener component from Firefly plug-in. 





























Virtual robot configuration and control: 1) network and IP address configuration for communication with mobile application. Assigning unique ports for each implemented functionality. 2) Routines for receiving streamed data which are relative and absolute position data for RE movements. 3) A virtual model of KUKA 6-axis industrial robot. 4) Code generation from movement command, tool setup and robot configuration by using KUKA|prc core component

Pilot Testing and Manipulation Task

A robot arm would be positioned over a table scattered with metallic objects in a predefined layout. The robot would be equipped with a magnetic gripper allowing for lifting metallic objects. The participant’s task would be to collect certain objects and avoid the rest of the objects. Such collection of objects should be tested in both formats; point wise and continuous trajectories. The user’s motivation would be working as quickly and accurately as possible by incentivizing short finishing time. In terms of task manipulation evaluation, the analysis can go one step further from human-robot interface and compare which method of collecting objects is more efficient. Such pilot study proves the feasibility of proposed system. It will be extended to usability test in next steps.

Experimental Results And Future Plan

Experimental Results And future plans

the proposed system supports the following functionalities which are not available on current existing robot teach pendant interfaces.

  • Leveraging tilt sensor for relative orientation

  • Robot controlling with multiple operators and possibility of turn-taking for task completion

  • Robot end-effector positioning through a sketch-based UI

  • Destination selection for robot end-effector with single tap

  • Variable scales for movement steps: Adjusting the distance traveled by the end effector on point wise and continuous trajectories.

  • More accuracy in movement by integration of vision system into the workflow: mapping the view of robot work space into the smartphone screen as guide for navigation through target positions.

  • Possibility of teleportation

Controlling the robot with developed user interface.
In addition to more development of the system and its integration into vision system, interface improvement, and controlling the actual robot with proposed UI, the most key requirement in future steps is usability test and user study.  To evaluate the effectiveness of this system as a robot controller in compare with existing robot teach pendants, user study is design based on table 2.

Study Design

A between-subject user study will be conducted. The independent variable as is mentioned in table 2 would be the user interface, varied by the mobile application and the teaching pendant. Participants would be using the interface to complete the same manipulation task which is fulfilled in pilot study. Such usability test will happen with a KRC4 robot arm.

Measures and Anticipated statistical analysis

To measure the outcomes of the collection task, we would use several objective and subjective measures. The objective measures would be targeting efficiency and errors. The subjective measures would be asking for user effort and ease of control in a form of a questionnaire.

A two-way analysis of variance (ANOVA) would be conducted with user interface types and, movement natures as input variables to detect significant differences in measures.


RoboCollab allows non-expert users to collaborate and interact more effectively with KUKA robots in manipulation tasks. Existing teach pendants as the most common user interfaces in robot operation have deficiencies in terms of teleportation, turn-taking in task completion by multiple operators, as well as RE positioning with natural intuitive hand gestures. Proposed sketch-based user interface which has also, utilized embedded sensors in the smartphone has provided solutions for mentioned issues.