Enforcing strict constraints on the number of processed units per transaction is a fundamental component of AI TRiSM (Trust, Risk, and Security…

The security engineer's statement is technically correct. The computational resources (e.g., GPU cycles, memory) and associated financial cost of processing a request with a Large Language Model (LLM) are directly proportional to the number of tokens in the prompt and the generated completion, not the number of requests. A traditional rate limit (e.g., 60 requests per minute) is ineffective because it treats a short 10-token query the same as a complex 4,000-token query, even though the latter is orders of magnitude more expensive to process. A token-based limit (e.g., 100,000 tokens per minute) provides a much more accurate and effective mechanism for managing resource consumption, preventing denial-of-service (DoS) attacks based on resource exhaustion, and controlling operational costs.

The statement posits a "False" option, which is incorrect. Denying the engineer's claim would ignore the fundamental operational principles of transformer-based models like LLMs. The primary driver of load is token count, making request-based limiting a crude and imprecise control that can be easily abused by sending a small number of very long, computationally expensive prompts.

1. OpenAI. (2024). Rate limits. OpenAI API Documentation. (The official documentation for one of the most widely used LLM services explicitly implements both requests-per-minute and tokens-per-minute limits, stating that "The rate limit is determined by the greater of the two limits at the time of the request." This dual approach confirms that request count alone is insufficient and token count is a critical metric for managing load). Retrieved from https://platform.openai.com/docs/guides/rate-limits

2. Google Cloud. (2024). Quotas and limits for Generative AI on Vertex AI. (Google's official documentation for its generative AI services also specifies quotas in both requests per minute and tokens per minute, demonstrating that token throughput is a primary metric for managing service capacity and preventing overload). Retrieved from https://cloud.google.com/vertex-ai/generative-ai/docs/quotas

3. Weidinger, L., et al. (2021). Taxonomy of Risks posed by Language Models. (Section 4.2 discusses the risk of resource-intensive generation, where malicious actors can cause high computational costs. This risk is directly tied to the amount of text processed/generated, i.e., tokens, making token-based limits a direct mitigation). arXiv:2112.04359.