Research · Study 01
Gaze-Based Robot Failure Detection
Reading human non-verbal behaviour to detect and anticipate robot failures
For robots, detecting and predicting failures as early as possible is vital to prevent damage and negative user experiences. I study human non-verbal behaviour — especially gaze patterns — to identify cues that signal when a robot failure is about to occur or has just occurred.
Key Takeaways
- User gaze behaviour can signal the onset of a robot failure.
- Gaze patterns are related to the type of failure the robot makes.
- A random forest classifier showed strong potential for detecting failures within a few seconds after they occur.
Demo
Overview
This line of work investigates the relationship between human gaze behaviour and robot failures. Using robot-assisted collaborative tasks (such as tangram puzzles), I explore how different failure types affect human perception and gaze dynamics, presenting gaze as a potential signal for robot error detection and recovery.
Method and System
Participants work with a robot on collaborative tasks while experiencing both executional and decisional failures at different task stages. Eye tracking (Pupil Labs) is synchronised with ROS, and real-time gaze features are extracted to characterise how people look at the robot, the workspace, and the task before, during, and after a failure.
What I Found
Executional failures led to increased gaze shifts toward the robot, while decisional failures resulted in less variability in gaze transitions — especially when occurring at the end of a task. These differences make gaze a reliable indicator of whether a failure occurred and what kind of failure it was, which in turn can inform appropriate recovery actions.
Why It Matters
Human-centred failure detection lets robots notice when something has gone wrong even when the failure is not directly observable from their own sensors. By reading the human's reaction, a robot can recover faster and preserve user trust during collaboration.