Neural Network Lexicon

In-Context Learning

Short Definition

In-Context Learning (ICL) is the ability of large language models to perform new tasks by using examples provided in the input prompt, without updating the model’s parameters.

The model learns the task behavior directly from the context of the prompt.

Definition

In-context learning occurs when a model receives examples of a task inside the input sequence and then generalizes the pattern to produce correct outputs.

Instead of updating model parameters through training, the model adapts its behavior dynamically through the context window.

Given a prompt containing examples:

Input: 2 + 2 → 4
Input: 3 + 5 → 8
Input: 6 + 1 →

The model predicts:

7

The model infers the task from the examples rather than from parameter updates.

Core Idea

In-context learning relies on the model’s ability to recognize patterns within the prompt.

The prompt acts as a temporary task specification.

Conceptually:

Prompt examples → internal pattern recognition → correct output

This behavior allows large models to solve new tasks with only a few examples.

Minimal Conceptual Illustration

Example classification task:

Positive: I love this movie
Negative: This film is terrible
Positive: The story was amazing
Negative:

The model predicts:

The acting was awful

as Negative, based on the pattern in the prompt.

Few-Shot vs Zero-Shot vs In-Context Learning

MethodDescription
Zero-Shottask described without examples
Few-Shotsmall number of examples provided
In-Context Learninggeneral term for learning from prompt context

Few-shot prompting is a form of in-context learning.

Mechanism

The exact mechanism behind in-context learning is still an active research topic.

Hypotheses include:

  • implicit gradient descent inside the model
  • pattern matching in latent space
  • retrieval of learned task structures

Transformers may simulate learning algorithms through attention mechanisms.

Mathematical View

A pretrained model represents a conditional distribution:

[
P(y \mid x, c)
]

Where:

  • (x) = input
  • (y) = output
  • (c) = context (prompt examples)

The context influences the model’s prediction behavior.

Importance in Large Language Models

In-context learning is one of the defining capabilities of modern LLMs.

It enables models to:

  • adapt to new tasks instantly
  • perform reasoning tasks
  • follow instructions
  • generalize from examples

This capability reduces the need for task-specific fine-tuning.

Limitations

In-context learning has several limitations.

Context Length Limits

Only a limited number of examples can fit in the context window.

Prompt Sensitivity

Small prompt changes can alter model behavior.

Inconsistent Generalization

Models may fail to reliably learn patterns from examples.

Applications

In-context learning is widely used in:

  • prompt engineering
  • few-shot reasoning tasks
  • coding assistants
  • language translation
  • structured data extraction

It is a core capability of modern AI systems.

Summary

In-context learning allows large language models to perform new tasks by learning from examples provided in the prompt. Rather than updating model parameters, the model uses contextual information to infer patterns and produce appropriate outputs, enabling flexible task adaptation.

Related Concepts