The best evaluated agent succeeded in only 23% of RoboQuest episodes, despite being able to perform most required actions when researchers supplied the hidden information. The failure analysis attributed only a minority of failures to execution and found that agents often stopped exploring before obtaining the evidence needed for task completion.
RoboQuest is a 10-task benchmark designed around information that is absent from an agent’s initial observations, according to an arXiv preprint published on October 7, 2026. Rather than relying only on those observations, a mobile manipulation agent must physically gather evidence before attempting the assigned goal.
The tasks divide that uncertainty into three categories: search, manipulation-based inspection and interactive testing. An agent might need to determine where a relevant object is, manipulate something to reveal an unobserved property or test an unfamiliar tool to discover its effect. It must then use the evidence to adapt its subsequent actions and decide for itself when to commit to task completion.
Researchers evaluated five frontier multimodal agents through a common visuomotor interface. They also tested a π₀.₅ policy fine-tuned on released demonstrations covering complete episodes. That fine-tuned policy “almost never” succeeded, the paper reports, although the supplied material does not give its exact success rate. It also does not identify the five agents, specify the number of evaluation episodes or provide task-by-task results.
Separate tests isolated the execution skills used in the benchmark and gave agents the information that would otherwise have to be discovered. Under those conditions, the agents could carry out most required actions. The failure analysis attributed only a minority of failures to execution, indicating that other aspects of the tasks accounted for most failures.
A recurring failure was premature commitment. Agents often stopped exploring and made decisions before observing the evidence required for task completion. Their attempts to investigate could also disturb the surroundings, and the systems rarely prevented those disturbances or repaired them afterward. Most models additionally struggled to learn through trial and error.
The benchmark targets agents’ ability to acquire missing task information through physical interaction in unfamiliar environments. An agent may possess the execution skills needed for a task when the hidden information is supplied yet still fail to search, inspect or test effectively. The findings remain preprint results, and the supplied report does not detail the robot hardware, operating speeds, episode duration or severity of the disturbances.