AI Scaling Laws

Extracted Attributes

• Nature

  Often follow power-law relationships, leading to smooth and predictable performance improvements with increased scale.

• Definition

  Predictive relationships in deep learning that describe how the performance of models, particularly large language models, scales with factors like model size, dataset size, and computational resources.

• Key Factors

  Model size (number of parameters), training data volume, computational resources.

• Predicted Outcome

  Model performance improvement and loss reduction.

• Notable Laws/Concepts

  Chinchilla Scaling Law (DeepMind), Second Era of Scaling Laws.

• Current Research Focus

  Accounting for inference costs in addition to training costs, and addressing potential diminishing returns.

Timeline

• DeepMind published "Training Compute-Optimal Large Language Models," introducing the Chinchilla Scaling Law, which challenged previous assumptions about LLM training. (Source: web_search_results)

  2022-03-00

• Research in "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws" highlighted the trade-off between training and inference costs, suggesting models trained longer than Chinchilla-Optimal could be more efficient. (Source: web_search_results)

  2023-00-00

• OpenAI released o1, a model that used post-training techniques to improve step-by-step reasoning and computer code writing, potentially influencing the 'second era of scaling laws'. (Source: web_search_results)

  2023-12-00

• xAI is constructing Colossus, the world's largest GPU supercomputer, in Memphis, powered by over 100,000 Nvidia Hopper chips, to train the Grok 3 model. This serves as a crucial real-world test of AI Scaling Laws. (Source: related_documents)

  Ongoing

Scale AI

Scale AI, Inc. is an American data annotation company based in San Francisco, California. It provides data labeling, model evaluation, and software to develop applications for artificial intelligence. The company’s research arm, the Safety, Evaluation and Alignment Lab, focuses on evaluating and aligning large language models (LLMs), including through initiatives such as Humanity's Last Exam, a benchmark designed to assess advanced AI systems on alignment, reasoning, and safety. Scale AI outsources data labeling through its subsidiaries, Remotasks, which focuses on computer vision and autonomous vehicles, and Outlier, which focuses on data annotation for LLMs. Scale AI's customers in the commercial sector have included Google, Microsoft, Meta, General Motors, OpenAI, and Time. The company also directly works with world governments, including the United States on multiple military-related projects, and with Qatar to improve the efficiency of its social programs.

Web Search Results

• What if A.I. Doesn't Get Much Better Than This? | The New Yorker

  The venture capitalist Anjney Midha similarly spoke of a “second era of scaling laws.” In December, OpenAI released o1, which used post-training techniques to make the model better at step-by-step reasoning and at writing computer code. Marcus agreed: “A fifty-billion-dollar market, maybe a hundred.” The linguistics professor Emily Bender, who co-authored a well-known critique of early language models, told me that “the impacts will depend on how many in the management class fall for the hype from the people selling this tech, and retool their workplaces around it.” She added, “The more this happens, the worse off everyone will be.” Such views have been portrayed as unrealistic—Nate Silver once replied to an Ed Zitron tweet by writing, “old man yells at cloud vibes”—while we readily accepted the grandiose visions of tech C.E.O.s. Maybe that’s starting to change.

• How Scaling Laws Drive Smarter, More Powerful AI

  NVIDIA and our third-party partners use cookies and other tools to collect and record information you provide as well as information about your interactions with our websites for performance improvement, analytics, and to assist in our marketing efforts. To opt out of non-cookie personal information "sales" / "sharing" for targeted advertising purposes, please visit the NVIDIA Preference Center. We have also opted you out of "sharing"/"sales" of personal information outside of cookies. We have also opted you out of "sharing"/"sales" of personal information outside of cookies which overrides at least one of your previous settings. To opt-out of the "sale" and "sharing" of personal information through cookies, you must opt-out of optional cookies using the toggles below.

• Scaling Up: How AI Month 2025 Brought Big Ideas to Life at Penn ...

  # Scaling Up: How AI Month 2025 Brought Big Ideas to Life at Penn Engineering While recent research suggests that scaling laws may have reached a point of diminishing returns, AI Month at Penn Engineering shows no signs of slowing down. And Penn Engineering alumni at firms as varied as Apple and AMD shared how AI is shaping their work today — and what students should know as they enter the field. AI Month also served to showcase the recently opened Amy Gutmann Hall, Penn Engineering’s new home for data science and AI. *Penn Engineering Today* and *Penn Today* also published a number of AI Month-exclusive stories and social media posts: * Penn Engineers First to Train AI at Lightspeed

• 2.4: Scaling Laws | AI Safety, Ethics, and Society Textbook

  In this section, we discuss how the performance of deep learning models has scaled according to parameter count and dataset size, both of which are primarily bottlenecked by the computational resources available. **Scaling laws in deep learning predict loss based on model size and dataset size.** In deep learning, power-law relationships exist between the model’s performance and other variables. Generative models such as large language models tend to follow regular scaling laws—as model size and training data increase in scale, performance improves smoothly and predictably in a relationship described by a power-law equation. **In AI, scaling laws describe how loss changes with model and dataset size.** We observed that the performance of a deep learning model scales according to the number of parameters and tokens—both shaped by the amount of compute used.

• Language Model Scaling Laws: Beyond Bigger AI Models in 2024

  At the heart of this evolution are scaling laws, which describe the relationships between a model’s performance and its key attributes, i.e. _size_ (number of parameters), _training data volume_, and _computational resources_. In March 2022, DeepMind published “Training Compute-Optimal Large Language Models,” introducing the Chinchilla Scaling Law. This groundbreaking research challenged previous assumptions about LLM training and became the field’s most widely cited scaling law. Research in `Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws` (2023) highlighted the trade-off between training and inference costs, suggesting that models trained longer than Chinchilla-Optimal could be more efficient when accounting for inference demands. This shift towards inference scaling has significant implications for AI development and deployment and suggests that future models require more computational resources during operation, potentially increasing costs and improving performance on complex tasks.