C1000-185 practice question 22 of 35

In the context of Generative AI (GenAI), various embedding models are used to represent textual data. Which of the following best describes the difference between Word2Vec, BERT, and Sentence-BERT embedding models?

Domain Access Governance and Certification

Answer options

A: Word2Vec captures both word and sentence meanings in a single vector space, BERT generates only word embeddings, and Sentence-BERT generates embeddings for entire documents.

B: Word2Vec captures contextual relationships between words, while BERT and Sentence-BERT generate sentence-level embeddings based on the overall document length.

C: Word2Vec creates static word embeddings, BERT generates dynamic embeddings based on context, and Sentence-BERT produces embeddings specifically optimized for sentence-level tasks like semantic similarity. Correct answer

D: Word2Vec uses a transformer architecture for embedding generation, whereas BERT and Sentence-BERT use neural networks to model context.

Correct Answer:

C. Word2Vec creates static word embeddings, BERT generates dynamic embeddings based on context, and Sentence-BERT produces embeddings specifically optimized for sentence-level tasks like semantic similarity.

Explanation

Word2Vec generates static embeddings, where each word in the vocabulary has a single, fixed vector representation regardless of its context. BERT (Bidirectional Encoder Representations from Transformers) produces dynamic, contextualized embeddings; the vector for a word is influenced by the other words in the sentence. Sentence-BERT (SBERT) is a modification of BERT specifically fine-tuned to create semantically meaningful sentence embeddings. It uses a siamese network structure, making its output vectors directly comparable for tasks like semantic similarity, clustering, and information retrieval, which is a known weakness when using raw BERT embeddings for such tasks.

Why Incorrect

A: Word2Vec is designed for word-level embeddings, not sentence meanings. BERT can represent sentences, and SBERT is not exclusively for entire documents. B: Word2Vec embeddings are static and non-contextual. BERT and SBERT embeddings are determined by sentence content, not overall document length. D: This statement incorrectly reverses the architectures. Word2Vec uses a shallow neural network, whereas BERT and SBERT are based on the Transformer architecture.

References

1. Mikolov

Chen

Corrado

& Dean

J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781. (This foundational paper on Word2Vec describes its architecture

which produces a single static vector per word).

2. Devlin

Chang

M. W.

Lee

& Toutanova

K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Volume 1 (Long and Short Papers)

pp. 4171–4186. https://doi.org/10.18653/v1/N19-1423. (Section 3.1 explains how BERT's architecture generates representations based on the entire context).

3. Reimers

& Gurevych

I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)

pp. 3982–3992. https://doi.org/10.18653/v1/D19-1410. (The abstract and introduction explicitly state that SBERT is designed to derive semantically meaningful sentence embeddings that can be compared using cosine similarity).

Keep practicing

Take the free C1000-185 practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test