Which option is a benefit of ongoing pre-training when fine-tuning a foundation model (FM)?

Ongoing pre-training, also known as continued pre-training or domain-adaptive pre-training, involves further training a general foundation model on a large corpus of domain-specific, unlabeled data. This process adapts the model's internal representations, vocabulary, and understanding to the nuances of the target domain (e.g., legal, medical, or financial text). By aligning the model with the specific data distribution of the target domain before task-specific fine-tuning, it achieves a better starting point, which consistently leads to improved performance and accuracy on downstream tasks within that domain.

1. Gururangan

S.

Marasović

A.

Swayamdipta

S.

Lo

K.

Beltagy

I.

Downey

D.

& Smith

N. A. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8342–8356). The paper's central thesis

summarized in the abstract and demonstrated in Section 4 ("Results")

is that "pretraining on data from the target domain (domain-adaptive pretraining) leads to performance gains." (DOI: https://doi.org/10.18653/v1/2020.acl-main.740)

2. Bommasani

R.

et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford University Center for Research on Foundation Models (CRFM). In Section 4.2.1 ("Adaptation")

the report discusses methods for adapting FMs

stating

"The goal of adaptation is to steer the behavior of a foundation model to better perform a desired downstream task." Continued pre-training is a key method to achieve this performance improvement. (Page 63).

3. Stanford University. (2023). CS224N: NLP with Deep Learning

Winter 2023 Lecture 12: Pretraining and Transfer Learning. The lecture notes explain that continued pretraining on a domain-specific corpus before fine-tuning helps the model learn the specific statistics and vocabulary of that domain

which improves final task performance.