An administrator is integrating an application with an endpoint and finds that the application is experiencing high latency. What action should the…

High latency in an application interacting with an endpoint is frequently caused by resource contention, where the endpoint cannot process the volume of incoming requests quickly enough. Increasing the number of endpoint instances (also known as replicas) enables horizontal scaling. This distributes the inference request load across multiple parallel instances. By doing so, it reduces queuing delays and allows the system to handle a higher degree of concurrency, which directly results in lower average latency per request and improved overall performance.

A. Reduce the payload size limit: This restricts the amount of data per request but does not inherently improve the endpoint's processing speed or reduce latency caused by high traffic.

B. Increase the timeout value on the endpoints: This action only makes the client application wait longer before failing a request; it does not reduce the actual processing time or fix the underlying latency issue.

C. Increase the cache expiration time: While caching can reduce latency for repeated requests, simply increasing the expiration time does not help with new, uncached requests or latency caused by processing load.

1. Nutanix AI Administration Guide v1.1, Chapter: Model Serving, Section: Deploying a Model. The guide explains that configuring the number of replicas for a model is essential "to handle the expected inference request load." Distributing load is a primary method for managing and reducing latency. The configuration options replicas, minReplicas, and maxReplicas are provided for this purpose.

2. Patterson, D. A., & Hennessy, J. L. (2017). Computer Organization and Design RISC-V Edition: The Hardware Software Interface. Morgan Kaufmann. Chapter 6, Parallel Processors from Client to Cloud. This foundational text explains that request-level parallelism, achieved by replicating servers (or instances) and using a load balancer, is a key strategy to decrease response time (latency) for services under high demand. This principle is directly applicable to scaling AI model endpoints.