In the context of fine-tuning LLMs, which of the following metrics is most commonly used to assess the performance of a fine-tuned model?

The primary goal of fine-tuning a Large Language Model (LLM) is to improve its performance on a specific, downstream task. To assess this performance and ensure the model generalizes well to new, unseen data, a validation set is used. Accuracy on this validation set is a standard and direct metric that measures how often the model's predictions are correct for the target task. This process helps in tuning hyperparameters and preventing overfitting, where the model performs well on training data but fails on real-world data.

A. Model size: This is an architectural characteristic of the model, not a direct measure of its performance on a specific task.

C. Training duration: This metric relates to computational efficiency and cost, not the qualitative or quantitative performance of the model's outputs.

D. Number of layers: This is a structural parameter of the neural network; while it influences model capacity, it does not directly measure task performance.

1. NVIDIA NeMo Framework Documentation: The NeMo documentation for fine-tuning models consistently emphasizes the use of validation and test datasets to evaluate performance. For tasks like text classification, accuracy is a key metric reported on the validation set during the training loop to monitor progress and select the best model checkpoint.

Source: NVIDIA NeMo Framework Documentation, "Text Classification (Sentiment Analysis) Tutorial," Section on "Training" and "Inference."

2. Stanford University CS231n Courseware: This course on deep learning explicitly details the methodology for training and evaluating models. It states that the validation set is crucial for tuning hyperparameters and serves as a proxy for the test set to evaluate model performance during development.

Source: Stanford CS231n: Convolutional Neural Networks for Visual Recognition, "Setting up the data and the model," Module 1, Section on "Data-driven approach, kNN, train/val/test splits."

3. Deep Learning by Goodfellow, Bengio, and Courville: This foundational textbook explains that machine learning model performance must be measured on a held-out test set (or a validation set during training). It distinguishes performance metrics (like accuracy) from model capacity parameters (like size or layers) and training costs (like duration).

Source: Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 5, "Machine Learning Basics," Section 5.2, "Performance Measures."