The Next Generation of World Models

Rudina Seseri

Last year, I discussed a class of AI known as a “world model,” an AI architecture that builds an internal representation of its surroundings to predict outcomes and guide decision-making, often bridging the gap between physical and digital worlds. Since then, the space has evolved at an exponential rate: Google DeepMind Mind announced Genie 3 last fall, Fei-Fei Li’s World Labs released Marble, Yann LeCun released LeWorldModel last month, and NVIDIA AI has further expanded its Cosmos world model.

In addition to new iterations of these models, some companies have also made acquisitions to continue advancing their technology, including Google, who acquired Glasswing Ventures’ portfolio company Common Sense Machines. We invested in CSM’s technology back in its seed round in 2020 because we saw the potential value in technology that helps to translate 2D into 3D models.

At the same time, world model development has not come without its challenges. I previously noted that overspecialization, complexity, and transparency are world models three largest limitations. I am excited to discuss Genie 3 and how it has evolved including in its recent announcement in May of this year because two of these limitations, overspecialization and complexity, are beginning to be addressed. These advancements are crucial in the path to making world models more viable and accessible in their use of training agents.

🗺️ What is Genie 3?

Genie 3 is a next-gen world model developed by Google DeepMind. Given a simple text prompt, it generates dynamic, explorable environments that a user can move around in and manipulate in real time. There is no hard-coded physics engine underneath. Instead, the system learns physics entirely through self-supervised learning, ingesting huge quantities of unlabeled video and developing its own internal model of how objects move.

While predecessor models (such as Genie 2) could sustain a generated environment for only seconds before falling apart, Genie 3 retains visual memory for approximately one minute and supports session lengths of several minutes, which is long enough to run meaningful training scenarios. It runs at 24 fps, about the speed of the average film, and its environments respond to user input in real time. The interactivity is an emergent property of the model’s training, not from any sort of explicit 3D modeling or scene reconstruction. DeepMind’s recent acquisition of CSM has helped bring that complementary capability directly into the team, and points toward where world models are heading next.

🤔 Why does Genie 3 matter, and what are its limitations?

For years, general-purpose AI has struggled with understanding “consequences.” An agent that can act in the real world, such a robot in a factory, needs to understand what happens when it does something, and that knowledge has historically needed to be acquired across a varied range of environments and tests. Providing that at scale has required physical hardware, which is expensive to build and slow to iterate. Genie 3 completely changes the economics involved by reducing the infrastructure required to achieve this training. By enabling AI agents to act inside generated worlds and observe how those worlds respond, it gives AI a place to learn causality without touching the real world, bringing us another step closer to AGI.

That said, Genie 3 still has limitations that will need to be addressed for broader world model development and for specific industry use cases:

🛠️ Applications of Genie 3

Genie 3’s strengths are best suited for use cases that require an agent to simulate, predict, and plan before acting in a real-world environment, such as:

Stay up-to-date on the latest AI news by subscribing to Rudina’s AI Atlas.