During the summer of 2019, I received an internship at X Inc., Google’s research and development subsidiary. While there, I worked on their Everyday Robot project, which aimed to build a fleet of mobile manipulator robots capable of operating in unstructured environments and performing human tasks. I worked closely with hardware, software, and machine learning engineers to accomplish these goals.

When I joined, the Everyday Robot team was focused on iterating and improving the design of their robots. I assisted in the software design and calibration of the perception system, facilitated machine learning experiments for training manipulation policies, and applied my expertise in motion planning to develop experiments to compare various manipulation pipelines.

I can personally attest to the amount of engineering effort it takes to implement a functioning and versatile pipeline based on methods from algorithmic robotics (like task and motion planning). To avoid this, the Everyday Robot team opted to place much of the project’s future on developing a basic visuomotor policy for manipulation. They trained this model using the existing fleet to sort trash into recycling, compose, or trash bins. Nowadays, many methods can achieve much more complex tasks in unstructured environments (see [1], [2], [3], [4], [5], [6], [7]). These new works use modern ML architectures like diffusion models and transformers, which fundamentally differ from the work done in 2019.

I have also noted that the robots from Everyday Robot have been used in several works incorporating large language models (LLMs) with vision policies to solve long-horizon tasks in unstructured environments [8], [9]. I’m excited to see what X, Google Brain, and Google DeepMind will get up to in the future using the robots developed through the Everyday Robot project!

I loved working with the team at X: they brought a vibrant energy that fueled my passion for innovation. I made some good friends whom I still keep up with!