NCA-GENL practice question 7 of 15
In the context of data preprocessing for Large Language Models (LLMs), what does tokenization refer to?
Full set: 95 questionsDomain Large Language Models (LLMs)
Tokenization is the foundational step in the Natural Language Processing (NLP) pipeline for Large Language Models (LLMs). It is the process of segmenting a raw text string into a sequence of smaller, manageable units called "tokens." These tokens can be as large as words or as small as characters. Modern LLMs commonly use subword tokenization algorithms (like BPE or WordPiece) which break words into smaller, meaningful parts. This allows the model to handle a large vocabulary efficiently, recognize morphological variations, and process words it has not seen during training (out-of-vocabulary words).
B. Converting text into numerical representations is known as embedding or vectorization, a process that occurs after tokenization.
C. Removing stop words (e.g., "the", "a", "is") is a specific text cleaning step, not the definition of tokenization itself.
D. Applying data augmentation is a technique to artificially expand the training dataset, which is distinct from the core text processing step of tokenization.
1. University Courseware:
Manning, C., & Jurafsky, D. (2023). Speech and Language Processing (3rd ed. draft). Stanford University. Chapter 2, Section 2.1 "Tokenization". This chapter explicitly defines tokenization as "the task of segmenting running text into words and sentences."
2. Official Vendor Documentation:
NVIDIA NeMo Toolkit Documentation. (n.d.). Tokenizers. Retrieved from the official NVIDIA docs. The documentation describes various tokenizers (e.g., Byte-Pair Encoding, WordPiece) and their function in "segmenting text into a list of tokens."
3. Peer-reviewed Academic Publications:
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A Survey of Large Language Models. arXiv preprint arXiv:2303.18223. Section 2.1.1, "Data Collection and Preprocessing," states: "The first step is to tokenize the raw text, which splits the text into a sequence of tokens." (https://doi.org/10.48550/arXiv.2303.18223)
Keep practicing
Take the free NCA-GENL practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other NCA-GENL practice questions
- Question 1What is the purpose of few-shot learning in prompt engineering?
- Question 2Which model deployment framework is used to deploy an NLP project, especially for high- p…
- Question 3Why do we need positional encoding in transformer-based models?
- Question 4What is Retrieval Augmented Generation (RAG)?
- Question 5Which technology will allow you to deploy an LLM for production application?
- Question 6In evaluating the transformer model for translation tasks, what is a common approach to a…
