Foundational Models

Extracted Attributes

• Aliases

  Large X model (LxM), Base model

• Coined by

  Stanford University's Center for Research on Foundation Models and Institute for Human-Centered Artificial Intelligence

• Definition

  Machine learning or deep learning model trained on vast datasets for a wide range of use cases

• Primary benefit

  Adaptability, reusability, accessibility for organizations

• Key technologies

  Deep neural networks, transfer learning, self-supervised learning

• Applications in fields

  Astronomy, radiology, genomics, coding, time-series forecasting, mathematics, chemistry

• Examples of text models

  OpenAI's GPT series, Google's BERT

• Examples of image models

  DALL-E, Stable Diffusion, Flamingo

• Examples of music models

  MusicGen, LLark

• Cost to build advanced models

  Hundreds of millions of dollars

• Examples of robotic control models

  RT-2, PaLM-E

Timeline

• Early examples of foundation models, such as OpenAI's GPT series and Google's BERT, began to emerge. (Source: Summary, Wikipedia)

  2018

• The term 'foundation model' was coined by researchers at Stanford University's Center for Research on Foundation Models and Institute for Human-Centered Artificial Intelligence. (Source: IBM, Ada Lovelace Institute)

  2021

• Development of foundation models continues across various modalities (text, images, music, robotics) and fields (astronomy, radiology, genomics, coding, etc.). (Source: Summary, Wikipedia)

  Ongoing

• Debate arises regarding the commercial viability of prioritizing AI safety in the 'AI Arms Race' to develop superintelligence and superior foundational models. (Source: Related Documents (All-In Podcast episode 185))

  Recent

Foundation model

In artificial intelligence (AI), a foundation model (FM), also known as large X model (LxM), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLM) are common examples of foundation models. Building foundation models is often highly resource-intensive, with the most advanced models costing hundreds of millions of dollars to cover the expenses of acquiring, curating, and processing massive datasets, as well as the compute power required for training. These costs stem from the need for sophisticated infrastructure, extended training times, and advanced hardware, such as GPUs. In contrast, adapting an existing foundation model for a specific task or using it directly is far less costly, as it leverages pre-trained capabilities and typically requires only fine-tuning on smaller, task-specific datasets. Early examples of foundation models are language models (LMs) like OpenAI's GPT series and Google's BERT. Beyond text, foundation models have been developed across a range of modalities—including DALL-E, Stable diffusion, and Flamingo for images, MusicGen and LLark for music, and RT-2 for robotic control. Foundation models are also being developed for fields like astronomy, radiology, genomics, coding, times-series forecasting, mathematics, and chemistry.

Web Search Results

• What are foundation models for AI? - Red Hat

  A foundation model is a type of machine learning (ML) model that is pretrained to perform a range of tasks. Until recently, artificial intelligence (AI) systems were specialized tools, meaning that an ML model would be trained for a specific application or single use case. The term foundation model (also known as a base model) entered our lexicon when experts began noticing 2 trends within the field of machine learning: [...] ## Why are foundation models beneficial for organizations to adopt? Foundation models provide accessibility and a level of sophistication within the realm of AI that many organizations do not have the resources to attain on their own. By adopting and building upon foundation models, companies can overcome common hurdles such as: Limited access to quality data: Foundation models provide a model built on data that most organizations don’t have access to.

• Foundation Models in Generative AI Explained - AWS

  Trained on massive datasets, foundation models (FMs) are large deep learning neural networks that have changed the way data scientists approach machine learning (ML). Rather than develop artificial intelligence (AI) from scratch, data scientists use a foundation model as a starting point to develop ML models that power new applications more quickly and cost-effectively. The term foundation model was coined by researchers to describe ML models trained on a broad spectrum of generalized and [...] Foundation models are a form of generative artificial intelligence (generative AI). They generate output from one or more inputs (prompts) in the form of human language instructions. Models are based on complex neural networks including generative adversarial networks (GANs), transformers, and variational encoders. [...] A unique feature of foundation models is their adaptability. These models can perform a wide range of disparate tasks with a high degree of accuracy based on input prompts. Some tasks include natural language processing (NLP), question answering, and image classification. The size and general-purpose nature of FMs make them different from traditional ML models, which typically perform specific tasks, like analyzing text for sentiment, classifying images, and forecasting trends.

• Foundation model - Wikipedia

  [edit] Technologically, foundation models are built using established machine learning techniques like deep neural networks, transfer learning, and self-supervised learning. Foundation models differ from previous techniques as they are general purpose models that function as a reusable infrastructure, instead of bespoke and one-off task-specific models. [...] - In the United Kingdom, the Competition and Markets Authority's AI Foundation Models: Initial Report defines foundations model as "a type of AI technology that are trained on vast amounts of data that can be adapted to a wide range of tasks and operations." [...] - In the European Union, the European Parliament's negotiated position on the E.U. AI Act defines a foundation model as an "AI model that is trained on broad data at scale, is designed for generality of output, and can be adapted to a wide range of distinctive tasks".

• What Are Foundation Models? - IBM

  My IBM Log in Subscribe # What are foundation models? ## Authors Rina Diane Caballar Staff Writer ## What are foundation models? Foundation models are artificial intelligence (AI) models trained on vast, immense datasets and can fulfill a broad range of general tasks. They serve as the base or building blocks for crafting more specialized applications. [...] In the realm of robotics, foundation models can help robots rapidly adapt to new environments and generalize across various tasks, scenarios and machine embodiments. For example, the PaLM-E embodied multimodal language model transfers knowledge from PaLM’s language and visual domains to robotics systems and is trained on robot sensor data.5 ### Software code generation [...] Researchers at Stanford University’s Center for Research on Foundation Models and Institute for Human-Centered Artificial Intelligence coined the term “foundation models” in a 2021 paper. They characterize these models as a “paradigm shift” and describe the reasoning behind their naming: “[A] foundation model is itself incomplete but serves as the common basis from which many task-specific models are built via adaptation. We also chose the term ‘foundation’ to connote the significance of

• What is a foundation model? - Ada Lovelace Institute

  ## AI technologies and foundation models ### What is a foundation model? Foundation models are AI models designed to produce a wide and general variety of outputs. They are capable of a range of possible tasks and applications, such as text, image or audio generation. They can be standalone systems or can be used as a ‘base’ for many other applications.5 [...] The term ‘foundation model’ was popularised in 2021 by researchers at the Stanford Institute for Human-Centered Artificial Intelligence, in collaboration with the Stanford Center for Research on Foundation Models, an interdisciplinary initiative set up by the Stanford Institute for Human-Centered AI . These researchers defined foundation models as ‘models trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.’7 [...] | Foundation model (see also ‘GPAI’) | Described by researchers at Stanford University Human-Centered Artificial Intelligence as: ‘AI neural network trained on broad data at scale that can be adapted to a wide range of tasks’ 39 40 | Coined by Stanford University Human-Centered Artificial Intelligence. ‘Foundation models are general-purpose technologies that function as platforms for a wave of AI applications, including generative AI: AI systems that can generate compelling text, images,