Interactive World Models

Output

Interactive 3D worlds (often referred to as 4D, including time)

Category

AI Technology

Function

Generates playable worlds from text

Mechanism

Reverse-engineers intuitive physics

Key Example

Sora (OpenAI, considered a world model)

Limitations

High development cost, technical hurdles (e.g., temporal consistency, practical action spaces)

Applications

Robotics, Embodied AI, Game Development, Virtual Photography, Training Perception AI, Autonomous Vehicles, Industrial Machines

Desired Properties

Structured, causal representations; identification of concepts; interactive; adaptable

In another example, Decart and Etched created Oasis as a technical demo of a Minecraft-like generative interactive world model. In a first for a publicly available demo, Oasis takes in user keyboard input and generates real-time, physics-based gameplay, allowing players to move, jump, pick up items, break blocks, and more. Its world model understands game elements like building, lighting, physics, and inventory management. [...] In machine learning terms, a world model can imagine how a virtual (or physical) world evolves in response to an agent’s (for example, a player’s) behavior. Building on advancements in video generation and autonomous vehicles, these “world simulators” can deliver three-dimensional and interactive experiences with temporal and spatial consistency—also considered “4D” (3D + time). [...] AI world models are gearing up to fundamentally reshape game development, driving innovations in how virtual environments are created, understood, and interacted with. This journey—from foundational breakthroughs to cutting-edge applications—shows a series of rapid advancements, each building on the successes of its predecessors. In the same way, AIs have become better at understanding and playing games, and they are now rapidly improving at visualizing and creating complex, interactive 3D

World models have immediate applications in physical AI, particularly in robotics and autonomous vehicles. These systems require interactive data that demonstrates how actions lead to specific outcomes, unlike traditional vision models that simply observe static data. [...] Since this work focuses solely on images, it lacks a time component. The world that the IWM models exists without a temporal dimension, limiting its application to real-world situations. Additionally, the action space isn’t particularly useful for practical applications. Nevertheless, this approach helps develop richer image representations than those from I-JEPA, which relies exclusively on masking. Distortions are often used to make the image representations invariant to them. However, in [...] The previous works focus on generating interactive environments that can be used to train intelligent autonomous agents. Now, we will explore approaches that use world models to develop the agents themselves. This is essential for leveraging all the generated environments. We want to create autonomous agents capable of performing well across diverse environments and tasks, especially with limited data. World models provide valuable solutions to this challenge.

“We already have the ability to create virtual, interactive worlds, but it costs hundreds and hundreds of millions of dollars and a ton of development time,” Johnson said. “[World models] will let you not just get an image or a clip out, but a fully simulated, vibrant, and interactive 3D world.” ## High hurdles While the concept is enticing, many technical challenges stand in the way. [...] OpenAI notes in a blog that Sora, which it considers to be a world model, can simulate actions like a painter leaving brush strokes on a canvas. Models like Sora — and Sora itself — can also effectively simulate video games. For example, Sora can render a Minecraft-like UI and game world. Future world models may be able to generate 3D worlds on demand for gaming, virtual photography, and more, World Labs co-founder Justin Johnson said on an episode of the a16z podcast.

World models can create more realistic and physically accurate visual content by understanding the underlying principles of how objects move and interact. These models can generate realistic 3D worlds on demand for many uses, including video games and interactive experiences. In certain cases, outputs from highly accurate world models can take the form of synthetic data, which can be leveraged for training perception AI. [...] When combined with large language models (LLMs), world models help AI understand instructions in natural language and interact more effectively. For example, a delivery robot could interpret a spoken request to "find the fastest route" and simulate different paths to determine the best one. This predictive intelligence makes physical AI models more efficient, adaptable, and safer—helping robots, autonomous vehicles, and industrial machines operate smarter in complex, real-world environments.