Scenario: An email filtering system needs to fine-tune a pre-trained BERT model for spam detection using a labeled email dataset (binary…

The standard and most effective method for adapting a pre-trained BERT model to a new classification task, such as spam detection, is through fine-tuning. This process involves loading the weights of the pre-trained model, which has learned general language representations. The original final layer, designed for pre-training tasks (like Masked Language Modeling), is discarded. A new, randomly initialized classification layer (the "head") suitable for the specific downstream task (e.g., a linear layer with a sigmoid or softmax function for binary classification) is added. The entire model, including the pre-trained layers and the new classifier, is then trained on the task-specific labeled dataset (e.g., spam emails), allowing all weights to be adjusted to the new domain.

A. This describes feature extraction, where only the new classifier is trained. Fine-tuning, which is more effective, involves updating the weights of the entire model, not just the final layer.

C. Attaching a classifier in parallel is an architecturally incorrect approach. The standard method is to replace the pre-training head with a task-specific head sequentially.

D. The scenario specifies a binary classification task (spam vs. not spam). Introducing a multi-task classifier is irrelevant and unsupported by the problem description, which only mentions a single labeled dataset.

1. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. In Section 4.1, the paper states: "For classification tasks... A simple softmax classifier is added on top of the transformer output... we simply plug in the task-specific inputs and outputs into BERT and fine-tune all the parameters end-to-end." This directly supports replacing the head and training the entire model.

2. Hugging Face. (n.d.). BertForSequenceClassification. Hugging Face Documentation. Retrieved from https://huggingface.co/docs/transformers/modeldoc/bert#transformers.BertForSequenceClassification. The documentation describes this class as "Bert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output)". This confirms the architecture of replacing the original head with a new classifier.

3. Manning, C., Jurafsky, D., & Jurafsky, D. (2023). Speech and Language Processing (3rd ed. draft). Stanford University. Chapter 10, Section 10.5 "Fine-tuning", describes the process: "The standard paradigm for using a pretrained model like BERT for a downstream task is to fine-tune it... we add a small task-specific architecture... and then jointly train the parameters of the new layers and the pretrained parameters of the transformer network". This validates discarding the old head, adding a new one, and training all parameters.