A company wants to use language models to create an application for inference on edge devices. The inference must have the lowest latency possible…

To achieve the lowest possible inference latency, processing must occur locally on the edge device, which eliminates network round-trip time inherent in API-based solutions. Small language models (SLMs) are specifically designed and optimized for resource-constrained environments like edge devices. Their smaller size and lower computational requirements result in significantly faster processing (lower computational latency) and a smaller memory footprint compared to large language models (LLMs). Therefore, deploying an optimized SLM directly on the edge device is the most effective strategy to meet the requirement for the lowest possible latency.

1. Academic Publication: Ghasemzadeh

P.

et al. (2024). A Survey on Large Language Models for Edge Intelligence: A New Horizon. arXiv preprint arXiv:2402.18231. In Section III.A

"Model Compression for On-Device LLMs

" the paper discusses that deploying large models on edge devices is challenging due to "high latency and energy consumption." It highlights the necessity of using smaller

compressed models for efficient on-device execution. (DOI: https://doi.org/10.48550/arXiv.2402.18231)

2. University Courseware: Stanford University

CS 329T: "On-device AI". Lecture 11

"On-device Inference

" details the primary motivations for on-device AI

listing "Latency: no network roundtrip" as a key advantage. The lecture materials emphasize that models must be small and computationally efficient to run effectively on mobile and edge hardware

directly supporting the use of SLMs over LLMs for this purpose.

3. Academic Publication: Zafrir

O.

et al. (2021). Prune Once for All: Sparse Pre-Trained Language Models. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021). This paper and similar research on model optimization demonstrate that creating smaller

sparser models (a characteristic of SLMs) is a critical technique for enabling "fast inference on resource-constrained hardware

" such as edge devices. (Available via NeurIPS proceedings).