A Generative Al Engineer is tasked with developing an application that is based on an open source large language model (LLM). They need a foundation LLM with a large context window. Which model fits this need?

The primary requirement is a foundation LLM with a large context window. Among the options provided, DBRX has the largest context window by a significant margin. DBRX is a state-of-the-art, open-source, mixture-of-experts (MoE) model from Databricks that was trained with a context length of 32,768 (32k) tokens. This extensive context window makes it highly suitable for applications that need to process and generate text based on long documents, detailed histories, or complex instructions, directly addressing the engineer's need.

A. DistilBERT: This is a smaller, distilled version of BERT with a maximum context window of only 512 tokens. It is also primarily an encoder model, not a generative one.
B. MPT-30B: The base MPT-30B model has a 2,048 (2k) token context window, with some fine-tuned variants extending to 8,192 (8k) tokens, which is still considerably smaller than DBRX.
C. Llama2-70B: The Llama 2 family of models was trained with a context length of 4,096 (4k) tokens. While larger than DistilBERT, this is only one-eighth the size of DBRX's context window.

1. DBRX: Databricks. (2024, March 27). Introducing DBRX: A New State-of-the-Art Open LLM. Databricks Blog. "DBRX has a 32k token context window". Link

2. Llama2-70B: Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta AI. Section 2.1, "Pretraining," states, "We adopt most of the pretraining setting and model architecture from Llama 1... we increase the context length to 4096." Link

3. MPT-30B: MosaicML. (2023, June 22). Introducing MPT-30B: A New Standard for Open-Source, Commercially Usable LLMs. Databricks Blog. "MPT-30B is a decoder-style transformer with 30B parameters. It was pretrained on 1T tokens of English text and code. The base model has a context window of 2k tokens." Link

4. DistilBERT: Sanh, V., et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Hugging Face. The official model card for distilbert-base-uncased specifies, "The maximum input sequence length is 512." Link