Benchmark Overview
Overview
The human-robot interaction (HRI) community has developed many methods for robots to navigate safely and socially alongside humans. However, experimental procedures to evaluate these works are usually constructed on a per-method basis. Such disparate evaluations make it difficult to compare the performance of such methods across the literature. To bridge this gap, we have constructed a simulated environment for social navigation are conducting a public benchmark: SEANavBench! SEANavBench is a collaboration between groups at Yale and Carnegie Mellon Universities and was jointly designed based on two works: SEAN (HAI '20) and SocNavBench (THRI '21).
Environments
SEANavBench comprises multiple environments with varying sizes, pedestrian velocities, and pedestrians densities.
Example environments in SEAN showing the diversity of locations and pedestrian densities
Task
Algorithms under evaluation (called "candidates") will be given a set of "tasks" which comprise an environment, a start location, and a goal location. The agent's aim will be to complete as many tasks as efficiently as possible, by traversing the environment from start to goal while navigating safely around pedestrians. SEANavBench will automatically record your navigation and evaluate it using our pre-defined metrics. Tasks will terminate after a pre-determined timeout which varies per task but guarantees reachability (on average, tasks have a ~2 minute timeout). Tasks will not be terminated on pedestrian collisions, but each collision will be recorded and reported.
Simulated robot details
Candidate algorithms will interface with the simulated robot (based on Kuri) via velocity control. The robot's linear & angular velocities are limited to a maximum of 2m/s & 0.5rad/s, respectively. The simulator runs as an asynchronous server, i.e. as a robot "thinks" simulated time will continue to elapse.
Pedestrian control
SEANavBench supports two methods of pedestrian control: algorithmic (via a pedestrian trajectory model) and replay-based (based on real-world pedestrian data). This variety in pedestrian data allows us to evaluate social navigation algorithms in different ways.
Pedestrian control: replay-based
In this mode, pedestrians paths are replayed from pedestrian trajectory datasets such as UCY & ETH. These data include varied crowd densities and walking speeds, as well as interesting pedestrian behaviors, such as grouping, following, passing, pacing, and waiting. Pedestrian trajectory data is replayed at the simulator tick rate with a 1:1 ratio to recorded time, i.e. replayed pedestrian trajectories have the same velocities as the recorded ones. If the simulated robot collides with a pedestrian, the benchmark registers that collision and the pedestrian’s replayed motion is unchanged.
Example recreation of a scene from the ETH Zurich pedestrian dataset
Pedestrian control: model-based
Pedestrians in SEANavBench can also be controlled using a social forces model allowing them to react to the robot's motion during simulation. In this mode, their starts and goals are sampled from a graph underlying each scene.
Evaluation and result reporting
Details on the evaluation procedure and metrics can be found here
Submission
Submission details are here