Context window

Key Mechanism

Self-attention

Security Risk

Vulnerability to data poisoning if windows are narrow or poorly managed

Measurement Unit

Tokens (representing words or parts of words)

Primary Function

Defines the amount of information a model can consider at one time

Core Architecture

Transformer

Llama 3.1 Capacity

128,000 tokens

Gemini 1.5 Pro Capacity

1,000,000 tokens

Mistral Large 2 Capacity

128,000 tokens

2024-02-15

2024-07-23

The context window (or “context length”) of a large language model (LLM) is the amount of text, in tokens, that the model can consider or “remember” at any one time. In real-world terms, the context length of a language model is measured not in words, but in *tokens.* To understand how context windows work in practice, it’s important to understand how these tokenswork. The notion of a context window is relevant to *any machine learning model that uses the* *transformer architecture**,* which comprises most modern generative AI models, including nearly all LLMs. Transformer models use a *self-attention* mechanism to calculate the relationships and dependencies between different parts of an input (like words at the beginning and end of a paragraph). Llama’s context length was significantly increased with the launch of Llama 3.1 models, which offered 128,000-token long context windows. Mistral Large 2, the flagship model offered by Mistral AI, has a context window of 128,000 tokens.

A context window is how information is entered into a large language model (LLM). **A context window** refers to the amount of information a large language model (LLM) can process in a single prompt. Before these breakthroughs in context windows, teams working with gen AI had to get creative with prompt engineering to make the most of their 1,500 words. * *Expanded training data sets.* New, long-context data sets help models learn to process more extensive texts and other forms of content, enhancing their ability to work with lengthier, more complex inputs. As researchers continue to rapidly expand context windows via novel model structures, more efficient long-context data sets, and innovative training techniques, the field of AI is pushing the boundaries of what can be achieved. Long context windows accelerate the already blistering pace of gen AI development, enabling models to process immense and diverse data sources—from expansive text collections to hours of multimedia.

When using extended thinking, all input and output tokens, including the tokens used for thinking, count toward the context window limit, with a few nuances in multi-turn situations. However, previous thinking blocks are automatically stripped from the context window calculation by the Claude API and are not part of the conversation history that the model "sees" for subsequent turns, preserving token capacity for actual conversation content. + The effective context window calculation becomes: `context_window = (input_tokens - previous_thinking_tokens) + current_turn_tokens`. The diagram below illustrates the context window token management when combining extended thinking with tool use:. All other previous blocks still count as part of the token window, and the thinking block in the current `Assistant` turn counts as part of the context window. + The effective context window calculation for extended thinking with tool use becomes: `context_window = input_tokens + current_turn_tokens`. See our model comparison table for a list of context window sizes and input / output token pricing by model.Extended thinking overview.

A context window in NLP refers to the number of words or tokens around a specific word that a machine learning model considers when trying to understand that word or generate a prediction. If the word under consideration is 'sat', and you have a context window of size 2, the model would consider the words 'The', 'cat', 'on', and 'the' as context for understanding 'sat'. In traditional N-gram models, a context window helps in predicting the next word in a sequence. In neural network-based models like Word2Vec or RNNs, the context window helps in embedding a word in a multi-dimensional space where words with similar context occupy close positions. Understanding the concept of a context window is particularly relevant when considering the security implications of AI and NLP models. A narrow or inappropriate context window can make the model vulnerable to data poisoning attacks, where an attacker introduces misleading data into the training set.