World Models

Extracted Attributes

Mental model

A mental model is an internal representation of external reality: that is, a way of representing reality within the mind. Such models are hypothesized to play a major role in cognition, reasoning and decision-making. The term for this concept was coined in 1943 by Kenneth Craik, who suggested that the mind constructs "small-scale models" of reality that it uses to anticipate events. Mental models can help shape behaviour, including approaches to solving problems and performing tasks. In psychology, the term mental models is sometimes used to refer to mental representations or mental simulation generally. The concepts of schema and conceptual models are cognitively adjacent. Elsewhere, it is used to refer to the mental model theory of reasoning developed by Philip Johnson-Laird and Ruth M. J. Byrne.

Web Search Results

• GitHub - nik-55/world-models: A curated list of research and projects on world models · GitHub

  BranchesTags Open more actions menu ## Folders and files | Name | Name | Last commit message | Last commit date | --- --- | | Latest commit History92 Commits 92 Commits | | LICENSE | LICENSE | | | | README.md | README.md | | | | | ## Repository files navigation A world model is a deep neural network system that learns to internally represent and simulate how the world works including its physical dynamics, objects, agents, and causal relationships so that it can predict how environments evolve and how actions will affect them. Instead of passively recognizing patterns, a world model builds an active understanding of change, enabling it to generate, imagine, and interact with coherent virtual worlds over time. [...] > > To create systems capable of understanding the world, we must move beyond training AI exclusively on written text. Machines must be able to observe their environments, understand physical interactions, anticipate outcomes, and reason about changes over time. World models are a step in that direction. They allow intelligent systems to form rich representations of environments and simulate possible futures before actions are taken. > > Rather than merely predicting the next token, world models aim to predict the consequences of actions and how environments evolve over time. A system that truly understands the world cannot rely on language alone. It must learn from the structure, motion, causality, and sensory richness of reality. > > Most of reality is never written down. > > — Source

• World model (artificial intelligence)

  A world model in artificial intelligence is a machine learning system that builds an internal representation of an environment. Often this is via understanding objects within video, which predictive LLMs cannot. The model predicts how that environment changes over time in response to actions. Researchers design world models to help agents plan, reason, and act without constant real-world trial and error. World models differ from systems that merely classify or generate outputs. They simulate dynamics such as physics, object interactions, and causality. Early ideas date to the 1990s. Modern versions power robots, autonomous driving, and interactive video generation. ## History [edit&action=edit&section=1 "Edit section: History")] [...] World models divide an inferencing task into work performed by encoders, predictors, simulators, and other pieces. They typically handle multimodal inputs such as video, lidar, radar, and audio, guided by textual prompting. LLMs power chatbots and code assistants. World models drive embodied agents that act in dynamic environments, such as autonomous driving. The two may be combined in hybrid systems. For example, a LLM handles instructions, while a world model manages low-level control. World model proponents such as LeCun claim that because LLMs are trained only on text, they have no ability to predict anything beyond text, such as real-world events. ## Benchmarks [edit&action=edit&section=5 "Edit section: Benchmarks")] [...] ## Architecture [edit&action=edit&section=2 "Edit section: Architecture")] World models process raw sensory data such as video frames or lidar scans. They compress this input into compact latent representations. The system then predicts future representations rather than pixel-by-pixel reconstructions. Many modern world models use joint embedding predictive architecture (JEPA). An encoder turns observations into embeddings. A predictor estimates one or a suite of embeddings from the current one and an action. In some cases a critic chooses one embedding as the best result. A regularizer keeps embeddings well-behaved.

• Medium

  Sign up Sign in Sign up Sign in Unknown user # Understanding World Models in AI: A Technical Guide Sulbha Jain -- 2 Listen Share Deep reinforcement learning and AI have made tremendous progress partly due to innovations like World Models. In the previous post introduced World Models. World models, which are AI systems that can use their understanding of the world to simulate aspects of it, enabling agents to predict both how an environment will evolve and how their actions will affect it. World models are a key stepping stone on the path to AGI, since they make it possible to train AI agents in an unlimited curriculum of rich simulation environments. [...] This guide covers key ideas about the architecture, purpose, and applications of World Models, as well as their relationship to agents, synthetic data, ChatGPT, and more. ### 1. What is the Purpose of World Models? World models aim to build an internal, compact, predictive representation of an environment’s dynamics — essentially a “mental map” that allows an agent to simulate future states and plan actions before executing them. This abstracts raw sensory inputs into latent features for efficient learning and better long-term strategy formulation.¹¹ ### 2. When to Use World Models? What is the Motivation?

• What Is a World Model? | NVIDIA Glossary

  1. Glossary 2. World Models ## What Is a World Model? World models are AI tools that understand the dynamics of the real world, including physics and spatial properties. They can use input data, including text, image, video, sound, and movement, to predict what happens next. ## How Are World Models Built? World models have gone from hand-coded rules to billion-parameter neural networks, but the underlying goal has stayed the same: Teach AI how to behave in the real world. [...] ### Robotics World models generate photorealistic synthetic data and predictive world states to help robots develop spatial intelligence. Using virtual simulations powered by physical simulators, these models let robots practice tasks safely and efficiently, accelerating learning through rapid testing and training. They help robots adapt to new situations by learning from diverse data and experiences. Modified world models enhance planning by simulating object interactions, predicting human behavior, and guiding robots to reach goals accurately. They also enhance decision-making by conducting multiple simulations and learning from the feedback. With virtual simulations, developers can reduce real-world testing risks, cutting time, costs, and resources. ### Video Analytics [...] Accelerates AI Model Training Starting from a pretrained world foundation model means developers don’t build from scratch and they inherit physics understanding, spatial reasoning, and temporal coherence, then post-train on domain-specific data to reach production performance faster. ## What Are the Real-World Applications of World Models? World models, when used with 3D simulators, serve as virtual environments to safely streamline and scale training for autonomous machines. With the ability to generate, curate, and encode video data, developers can better train autonomous machines to sense, perceive, and interact with dynamic surroundings. ### Autonomous Vehicles

• 'World Models,' an Old Idea in AI, Mount a Comeback

  Nash Weerasekera for Quanta Magazine ## Introduction The latest ambition of artificial intelligence research — particularly within the labs seeking “artificial general intelligence,” or AGI — is something called a world model: a representation of the environment that an AI carries around inside itself like a computational snow globe. The AI system can use this simplified representation to evaluate predictions and decisions before applying them to its real-world tasks. The deep learning luminaries Yann LeCun (of Meta), Demis Hassabis (of Google DeepMind) and Yoshua Bengio (of Mila, the Quebec Artificial Intelligence Institute) all believe world models are essential for building AI systems that are truly smart, scientific and safe.