Foundation models

Extracted Attributes

• Type

  Artificial Intelligence Model

• Impact

  Transformed AI system development, democratized AI

• Aliases

  Large X model (LxM), Pre-trained model

• Definition

  Machine learning or deep learning models trained on vast datasets, enabling them to be adapted across a wide range of use cases.

• Cost to adapt

  Significantly less expensive, typically requiring only fine-tuning on smaller, task-specific datasets

• Cost to build

  Hundreds of millions of dollars (for most advanced models)

• Key technologies

  Deep neural networks, transfer learning, self-supervised learning

• Primary function

  General-purpose technologies that function as platforms for a wave of AI applications, including generative AI

• Typical parameters

  At least 1,000,000,000 parameters (as per proposed AI Foundation Model Transparency Act of 2023)

Timeline

• Foundation models, also known as large X models (LxM), began transforming AI system development. (Source: summary, dbpedia)

  2018

• The term 'foundation model' was popularized by researchers at the Stanford Institute for Human-Centered Artificial Intelligence (HAI), in collaboration with the Stanford Center for Research on Foundation Models (CRFM). (Source: web_search_results, dbpedia)

  2021

• The AI Foundation Model Transparency Act of 2023 was proposed by U.S. House Representatives Don Beyer and Anna Eshoo, defining foundation models and outlining potential risks. (Source: web_search_results)

  2023

Foundation model

In artificial intelligence (AI), a foundation model (FM), also known as large X model (LxM), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLM) are common examples of foundation models. Building foundation models is often highly resource-intensive, with the most advanced models costing hundreds of millions of dollars to cover the expenses of acquiring, curating, and processing massive datasets, as well as the compute power required for training. These costs stem from the need for sophisticated infrastructure, extended training times, and advanced hardware, such as GPUs. In contrast, adapting an existing foundation model for a specific task or using it directly is far less costly, as it leverages pre-trained capabilities and typically requires only fine-tuning on smaller, task-specific datasets. Early examples of foundation models are language models (LMs) like OpenAI's GPT series and Google's BERT. Beyond text, foundation models have been developed across a range of modalities—including DALL-E and Flamingo for images, MusicGen for music, and RT-2 for robotic control. Foundation models are also being developed for fields like astronomy, radiology, genomics, music, coding, times-series forecasting, mathematics, and chemistry.

Web Search Results

• Foundation Models in Generative AI Explained

  Foundation models are a form of generative artificial intelligence (generative AI). They generate output from one or more inputs (prompts) in the form of human language instructions. Models are based on complex neural networks including generative adversarial networks (GANs), transformers, and variational encoders. [...] A unique feature of foundation models is their adaptability. These models can perform a wide range of disparate tasks with a high degree of accuracy based on input prompts. Some tasks include natural language processing (NLP), question answering, and image classification. The size and general-purpose nature of FMs make them different from traditional ML models, which typically perform specific tasks, like analyzing text for sentiment, classifying images, and forecasting trends. [...] are large deep learning neural networks that have changed the way data scientists approach machine learning (ML). Rather than develop artificial intelligence (AI) from scratch, data scientists use a foundation model as a starting point to develop ML models that power new applications more quickly and cost-effectively. The term _foundation model_ was coined by researchers to describe ML models trained on a broad spectrum of generalized and unlabeled data and capable of performing a wide

• Foundation Models: The Benefits, Risks, and Applications - V7 Labs

  Foundation models, also known as pre-trained models, are large-scale artificial intelligence (AI) models trained on vast amounts of data to acquire a deep understanding of language, images, or other modalities. These models serve as a starting point for various AI tasks, as they have already learned valuable data representations and can be fine-tuned or adapted for specific applications. overview of foundation model for decision making Source [...] With their ability to understand language, images, or multimodal data at a deep level, foundation models have paved the way for cutting-edge AI applications and accelerated development timelines. From language models like GPT and BERT to vision models like ResNet, foundation models excel in various domains, serving as starting points for specialized tasks. [...] Democratization of AI. Foundation models have already contributed to the democratization of AI by providing accessible, pre-trained models to developers and researchers. This trend is likely to continue, making AI more accessible to a wider audience. Open-source initiatives, collaborations, and knowledge sharing will drive the adoption and refinement of foundation models, allowing individuals and organizations with limited resources to leverage state-of-the-art AI capabilities.

• Foundation model

  [edit] Technologically, foundation models are built using established machine learning techniques like deep neural networks, transfer learning, and self-supervised learning. Foundation models differ from previous techniques as they are general purpose models function as a reusable infrastructure, instead of bespoke and one-off task-specific models. [...] Foundation models are built by optimizing a training objective(s), which is a mathematical function that determines how model parameters are updated based on model predictions on training data. Language models are often trained with a next-tokens prediction objective, which refers to the extent at which the model is able to predict the next token in a sequence. Image models are commonly trained with contrastive learning or diffusion training objectives. For contrastive learning, images are [...] - In the United States, the proposed AI Foundation Model Transparency Act of 2023 by House Representatives Don Beyer (D, VA) and Anna Eshoo (D, CA) defines a foundation model as "an artificial intelligence model trained on broad data, generally uses self supervision, generally contains at least 1,000,000,000 parameters, is applicable across a wide range of contexts, and exhibits, or could be easily modified to exhibit, high levels of performance at tasks that could pose a serious risk to

• What is a foundation model?

  ‘Foundation models are general-purpose technologies that function as platforms for a wave of AI applications, including generative AI: AI systems that can generate compelling text, images, videos, speech, music, and more’. ( [...] AI technologies and foundation models ------------------------------------- ### What is a foundation model? Foundation models are AI models designed to produce a wide and general variety of outputs. They are capable of a range of possible tasks and applications, such as text, image or audio generation. They can be standalone systems or can be used as a ‘base’ for many other applications.( [...] The term ‘foundation model’ was popularised in 2021 by researchers at the Stanford Institute for Human-Centered Artificial Intelligence, in collaboration with the Stanford Center for Research on Foundation Models, an interdisciplinary initiative set up by the Stanford Institute for Human-Centered AI . These researchers defined foundation models as ‘models trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.’(

• What Are Foundation Models?

  My IBM Log in Subscribe # What are foundation models? ## Authors Rina Diane Caballar Staff Writer ## What are foundation models? Foundation models are artificial intelligence (AI) models trained on vast, immense datasets and can fulfill a broad range of general tasks. They serve as the base or building blocks for crafting more specialized applications. [...] Researchers at Stanford University’s Center for Research on Foundation Models and Institute for Human-Centered Artificial Intelligence coined the term “foundation models” in a 2021 paper. They characterize these models as a “paradigm shift” and describe the reasoning behind their naming: “[A] foundation model is itself incomplete but serves as the common basis from which many task-specific models are built via adaptation. We also chose the term ‘foundation’ to connote the significance of [...] Their flexibility and massive size set them apart from traditional machine learning models, which are trained on smaller datasets to accomplish specific tasks, such as object detection or trend forecasting. Foundation models, meanwhile, employ transfer learning to apply the knowledge learned from one task to another. This makes them fit for more expansive domains, including computer vision, natural language processing (NLP) and speech recognition.

A foundation model is a large artificial intelligence model trained on a vast quantity of unlabeled data at scale (usually by self-supervised learning) resulting in a model that can be adapted to a wide range of downstream tasks. Foundation models have helped bring about a major transformation in how AI systems are built since their introduction in 2018. Early examples of foundation models were large pre-trained language models including BERT and GPT-3. Using the same ideas, domain specific models using sequences of other kinds of tokens, such as medical codes, have been built as well. Subsequently, several multimodal foundation models have been produced including DALL-E, Flamingo, and Florence. The Stanford Institute for Human-Centered Artificial Intelligence's (HAI) Center for Research on Foundation Models (CRFM) popularized the term.