What is "in-context learning" in the realm of Large Language Models (LLMs)?

In-context learning is the ability of a Large Language Model (LLM) to learn a new task at inference time based solely on the information provided in the input prompt. Instead of updating the model's weights through retraining or fine-tuning, the user provides a prompt containing a few examples (few-shot) or just instructions (zero-shot) of the desired task. The model uses this "context" to understand the task and generate the appropriate response for a new query within the same prompt.

A. Training a model on a diverse range of tasks is known as pre-training or multi-task training, not in-context learning.
B. Modifying the model's behavior permanently involves updating its weights through a process called fine-tuning. In-context learning is temporary.
C. Zero-shot learning is a specific type of in-context learning, but providing examples (few-shot) is the more general and powerful demonstration of this capability.

1. Brown

T. B.

Mann

B.

Ryder

N.

et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165. Section 2.1

"In-context Learning". "We use the term 'in-context learning' to refer to a model’s ability to learn a task from a few examples given in the prompt."

2. Stanford University. (2023). On the Opportunities and Risks of Foundation Models. Center for Research on Foundation Models (CRFM). Section 2.2

"Emergence". "In-context learning is an emergent capability of foundation models where the model learns to perform a task after being provided with a prompt that contains a small number of examples."