How do Large Language Models (LLMs) handle the trade-off between model size, data quality, data size and performance?

Large Language Models (LLMs) operate based on scaling laws, which describe the relationship between model performance, model size (number of parameters), and the amount of training data (number of tokens). Research has shown that for optimal performance at a given computational budget, these factors must be scaled in proportion to each other. Simply increasing model size without a corresponding increase in high-quality training data leads to diminishing returns and inefficient training. Therefore, achieving state-of-the-art results requires a careful balance of model size, data size, data quality, and the computational resources for training.

A. Prioritizing only model size is inefficient. A smaller model trained on more data can outperform a larger model trained on insufficient data.
B. Increasing only the number of tokens for a fixed model size will eventually lead to the model being undertrained and unable to learn effectively.
C. Disregarding model size is not feasible. The model's capacity, determined by its size, is a critical factor in its ability to learn from data.

1. Hoffmann

J.

Borgeaud

S.

et al. (2022). Training Compute-Optimal Large Language Models. DeepMind. arXiv:2203.15556 [cs.CL]. Section 3

"We find that for compute-optimal training

the model size and the number of training tokens should be scaled equally."

2. Oracle. (2024). OCI Generative AI. OCI Documentation. "The quality and performance of a generative AI model depend on several factors

including the model architecture

the size of the model (number of parameters)

and the quality and quantity of the data it was trained on."