In the context of Generative AI (GenAI), various embedding models are used to represent textual data. Which of the following best describes the…

Word2Vec generates static embeddings, where each word in the vocabulary has a single, fixed vector representation regardless of its context. BERT (Bidirectional Encoder Representations from Transformers) produces dynamic, contextualized embeddings; the vector for a word is influenced by the other words in the sentence. Sentence-BERT (SBERT) is a modification of BERT specifically fine-tuned to create semantically meaningful sentence embeddings. It uses a siamese network structure, making its output vectors directly comparable for tasks like semantic similarity, clustering, and information retrieval, which is a known weakness when using raw BERT embeddings for such tasks.

A: Word2Vec is designed for word-level embeddings, not sentence meanings. BERT can represent sentences, and SBERT is not exclusively for entire documents.

B: Word2Vec embeddings are static and non-contextual. BERT and SBERT embeddings are determined by sentence content, not overall document length.

D: This statement incorrectly reverses the architectures. Word2Vec uses a shallow neural network, whereas BERT and SBERT are based on the Transformer architecture.

1. Mikolov

T.

Chen

K.

Corrado

G.

& Dean

J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781. (This foundational paper on Word2Vec describes its architecture

which produces a single static vector per word).

2. Devlin

J.

Chang

M. W.

Lee

K.

& Toutanova

K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Volume 1 (Long and Short Papers)

pp. 4171–4186. https://doi.org/10.18653/v1/N19-1423. (Section 3.1 explains how BERT's architecture generates representations based on the entire context).

3. Reimers

N.

& Gurevych

I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)

pp. 3982–3992. https://doi.org/10.18653/v1/D19-1410. (The abstract and introduction explicitly state that SBERT is designed to derive semantically meaningful sentence embeddings that can be compared using cosine similarity).