Foundation Models

Extracted Attributes

• Definition

  Machine learning or deep learning models trained on vast datasets for wide applicability.

• Modalities

  Text, images, music, robotic control

• Adaptation Cost

  Considerably less expensive than development

• Development Cost

  Hundreds of millions of dollars for advanced models

• Alternative Names

  Large X Models (LxMs)

• Application Fields

  Astronomy, radiology, genomics, coding, chemistry, natural language processing (NLP), question answering, image classification, time-series forecasting, mathematics.

• Key Characteristics

  Trained on massive datasets, resource-intensive development, adaptable, general-purpose, leverage pre-trained capabilities, reusable infrastructure.

• Underlying Technologies

  Deep neural networks, transfer learning, self-supervised learning, generative adversarial networks (GANs), transformers, variational encoders.

Timeline

• The term 'foundation model' was popularized by researchers at the Stanford Institute for Human-Centered Artificial Intelligence, in collaboration with the Stanford Center for Research on Foundation Models. (Source: Web Search Results)

  2021

Foundation model

In artificial intelligence (AI), a foundation model (FM), also known as large X model (LxM), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLM) are common examples of foundation models. Building foundation models is often highly resource-intensive, with the most advanced models costing hundreds of millions of dollars to cover the expenses of acquiring, curating, and processing massive datasets, as well as the compute power required for training. These costs stem from the need for sophisticated infrastructure, extended training times, and advanced hardware, such as GPUs. In contrast, adapting an existing foundation model for a specific task or using it directly is far less costly, as it leverages pre-trained capabilities and typically requires only fine-tuning on smaller, task-specific datasets. Early examples of foundation models are language models (LMs) like OpenAI's GPT series and Google's BERT. Beyond text, foundation models have been developed across a range of modalities—including DALL-E and Flamingo for images, MusicGen for music, and RT-2 for robotic control. Foundation models are also being developed for fields like astronomy, radiology, genomics, music, coding, times-series forecasting, mathematics, and chemistry.

Web Search Results

• What are Foundation Models? - Generative AI - AWS

  Foundation models are a form of generative artificial intelligence (generative AI). They generate output from one or more inputs (prompts) in the form of human language instructions. Models are based on complex neural networks including generative adversarial networks (GANs), transformers, and variational encoders. [...] A unique feature of foundation models is their adaptability. These models can perform a wide range of disparate tasks with a high degree of accuracy based on input prompts. Some tasks include natural language processing (NLP), question answering, and image classification. The size and general-purpose nature of FMs make them different from traditional ML models, which typically perform specific tasks, like analyzing text for sentiment, classifying images, and forecasting trends. [...] are large deep learning neural networks that have changed the way data scientists approach machine learning (ML). Rather than develop artificial intelligence (AI) from scratch, data scientists use a foundation model as a starting point to develop ML models that power new applications more quickly and cost-effectively. The term _foundation model_ was coined by researchers to describe ML models trained on a broad spectrum of generalized and unlabeled data and capable of performing a wide

• Foundation model - Wikipedia

  [edit] Technologically, foundation models are built using established machine learning techniques like deep neural networks, transfer learning, and self-supervised learning. Foundation models differ from previous techniques as they are general purpose models function as a reusable infrastructure, instead of bespoke and one-off task-specific models. [...] Light Dark This page is always in light mode. From Wikipedia, the free encyclopedia Artificial intelligence model paradigm In artificial intelligence (AI), a foundation model (FM), also known as large X model (LxM), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLM) are common examples of foundation models. [...] Foundation models are built by optimizing a training objective(s), which is a mathematical function that determines how model parameters are updated based on model predictions on training data. Language models are often trained with a next-tokens prediction objective, which refers to the extent at which the model is able to predict the next token in a sequence. Image models are commonly trained with contrastive learning or diffusion training objectives. For contrastive learning, images are

• What is a foundation model? - Ada Lovelace Institute

  ‘Foundation models are general-purpose technologies that function as platforms for a wave of AI applications, including generative AI: AI systems that can generate compelling text, images, videos, speech, music, and more’. ( [...] AI technologies and foundation models ------------------------------------- ### What is a foundation model? Foundation models are AI models designed to produce a wide and general variety of outputs. They are capable of a range of possible tasks and applications, such as text, image or audio generation. They can be standalone systems or can be used as a ‘base’ for many other applications.( [...] The term ‘foundation model’ was popularised in 2021 by researchers at the Stanford Institute for Human-Centered Artificial Intelligence, in collaboration with the Stanford Center for Research on Foundation Models, an interdisciplinary initiative set up by the Stanford Institute for Human-Centered AI . These researchers defined foundation models as ‘models trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.’(

• Foundation Models for Natural Language Processing - SpringerLink

  This open access book gives a deep overview of pre-trained language foundation models (BERT, GPT) and state-of-the-art models for a wide range of NLP tasks.

• Foundation Models for Natural Language Processing -- Pre-trained ...

  | ACM classes: | I.2.6; I.2.7; I.2.8; I.2.10; I.4.8; I.4.10; I.5.2; I.5.4; I.7.0; J.1; J.3; K.4.1; K.4.2; K.5.0 | | Cite as: | arXiv:2302.08575 [cs.CL] | | | (or arXiv:2302.08575v1 [cs.CL] for this version) | | | Focus to learn more arXiv-issued DOI via DataCite | [...] Cornell University arxiv logo Help | Advanced Search arXiv logo Cornell University Logo ## quick links # Computer Science > Computation and Language # Title:Foundation Models for Natural Language Processing -- Pre-trained Language Models Integrating Media [...] | | | | --- | --- | | Comments: | This book has been accepted by Springer Nature and will be published as an open access monograph. this https URL. It is licensed under the CC BY-NC-SA license (this https URL), except for the material included from other authors, which may have different licenses | | Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM) | | MSC classes: | 68W20, 68W25 |