What are tokens in the context of generative AI models?

In generative AI, particularly in Large Language Models (LLMs), a token is the fundamental atomic unit of text that the model processes. Raw text input is first segmented into a sequence of these tokens through a process called tokenization. A token can represent a whole word (e.g., "apple"), a subword (e.g., the "token" and "ization" in "tokenization"), or even a single character or punctuation mark. This approach allows the model to handle a vast and open vocabulary efficiently, including rare words, misspellings, and compound words, by breaking them down into known smaller pieces. The model's input and output are fundamentally sequences of these tokens.

1. OpenAI Documentation: In the "Platform" documentation

under the section "Introduction to tokens

" it is stated: "Large language models process text by breaking it down into smaller units

called tokens. Tokens can be words

chunks of words

or single characters." This directly supports that tokens are the basic units of input/output.

Source: OpenAI. (n.d.). What are tokens?. OpenAI Platform Documentation. Retrieved from the official OpenAI website's section on tokens.

2. Stanford University Courseware (CS224N): The lecture notes for "CS224N: Natural Language Processing with Deep Learning" describe the initial step of any NLP pipeline as tokenization. It defines this process as "segmenting text into units such as words

punctuation

etc." These units are referred to as tokens.

Source: Manning

C. D.

& Jurafsky

D. (2023). Lecture 2: Word Vectors and Word Senses. Stanford CS224N: NLP with Deep Learning

Winter 2023.

3. Google Cloud AI Documentation: In the introduction to Large Language Models

the documentation explains: "LLMs break down text into smaller units called tokens for processing. A token can be a word

a subword

a character

or a group of characters."

Source: Google Cloud. (n.d.). Large language models introduction. Vertex AI Documentation.

4. Hugging Face Courseware: The official Hugging Face NLP course explains tokenization in detail: "Splitting the input into words

subwords

or symbols that are called tokens is called tokenization... The tokenizer has to convert our text inputs into numerical data the model can make sense of."

Source: Hugging Face. (n.d.). Chapter 2: Behind the pipeline - What is a Tokenizer?. Hugging Face NLP Course.