Last Updated: 9 November 2024
MIT researchers are pioneering a fresh approach to robot training, drawing inspiration from large language models. Their new method, dubbed Heterogeneous Pretrained Transformers (HPT), integrates varied data sources to equip robots with an expanded set of skills. Unlike traditional methods that rely heavily on task-specific data, HPT offers a quicker, more cost-effective, and versatile way to prepare robots for diverse tasks and environments.
This approach aims to break down complex training processes into more understandable steps, providing greater context to each element involved. By using diverse data sources, HPT allows readers to appreciate how varied inputs contribute to a robot's adaptability, making the discussion more accessible compared to traditional explanations.
Traditionally, the process of training robots has been meticulous, involving the gathering of specific data tailored to a particular robot and task within controlled settings. This approach, while precise, often lacks flexibility and incurs significant time and financial costs. Inspired by the training methods of language models like GPT-4, MIT's researchers have charted a novel path with HPT.
HPT merges various types of data—including camera images, language instructions, depth maps, and sensor signals—into a cohesive framework that a generative AI can interpret.
By leveraging data from simulations, real robots, and human demonstrations, HPT enables robots to learn new skills without the need for repeated foundational training. This makes robot training more accessible and adaptable, allowing them to better handle unfamiliar tasks and environments.
At the heart of HPT is a transformer model, similar to the ones powering large language models. This machine-learning transformer processes inputs from vision and proprioception to develop a universal model applicable to various robots and settings. The researchers pretrained their model using 52 datasets comprising over 200,000 robot trajectories across different scenarios, from simulations to human demonstration videos.
A major challenge was ensuring the transformer could handle a myriad of data types and sources. The solution lay in converting all inputs into a consistent type of token, enabling the transformer to process diverse information seamlessly.
This approach simplifies how data is used, making the process clearer for readers. With exposure to more data, the model's performance continuously improves, allowing it to generalize across different robots and tasks more efficiently than previous methods.
In experimental trials, the HPT approach surpassed traditional training methods by over 20% in both simulated and real-world scenarios. Even when faced with tasks unlike those encountered during training, robots using the HPT model showcased remarkable adaptability.
A key element of this success is the model's treatment of proprioception—robots' awareness of their movements and positioning. By giving equal weight to proprioception and visual data, the system can execute more complex, dexterous movements. This balanced focus empowers robots with a heightened understanding of their capabilities, leading to enhanced task performance.
By explaining how proprioception and visual data work together, this section provides greater context on the significance of these elements, allowing readers to understand their role in advancing robot capabilities.
The MIT team, led by Lirui Wang, envisions further improvements for HPT, especially in processing unlabeled data like advanced language models do.
“In robotics, people often claim that we don’t have enough training data. But in my view, another big problem is that the data come from so many different domains, modalities, and robot hardware. Our work shows how you’d be able to train a robot with all of them put together,”
Lirui Wang, Electrical Engineering & Computer Science, author of a paper on this technique.
Their long-term goal is to create a "universal robot brain"—a flexible intelligence that could be downloaded and used by any robot without additional training. Although this vision is still in the early stages, the team believes that scaling HPT could lead to breakthroughs similar to those seen in language AI.
This vision highlights the potential for creating a universal intelligence that could simplify robot training and deployment across industries, helping readers understand the long-term significance of HPT.
Supported by funding from the Amazon Greater Boston Tech Initiative and the Toyota Research Institute, this groundbreaking work was presented at the Conference on Neural Information Processing Systems, heralding a new era in general-purpose robotics.
