This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

If "to err is human," you could say robots are superhuman. They're more error-prone than we are, even when tasked with things like home and factory chores, which we might consider easy. One reason for these mistakes is that robots struggle to extract relevant details from the past and act on them in real time, even with vision-language models (VLMs), which interpret visual inputs such as videos and images along with natural language.

Recent approaches use AI agents, which are semiautonomous programs based on VLMs, that, in this case, decide when robots should interpret specific details from a long history of camera frames. However, this can lead to errors, high compute costs and poor reactivity.

A new approach called "Workspace Models" from researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and Carnegie Mellon University's Machine Learning Department (MLD) reformulates the agentic pipeline used to compress history for robots.

Instead of relying on a VLM to search through data on the spot, these large VLMs do the reasoning offline. During that process, these models teach the robot how to memorize key details by training a smaller module (also called the Workspace Model), which enables robotic arms, for example, to plan their next move in milliseconds in response to changes in their environment. The work is published on the arXiv preprint server.

Workspace Models learn how to recall key details by essentially converting a history of observations into high-dimensional data points, called "latent vectors," which encode information relevant for predicting future actions.

Source link