You have deployed a model on Vertex AI for real-time inference. During an online prediction request, you get an “Out of Memory” error. What should you do?

An "Out of Memory" (OOM) error during online prediction on Vertex AI indicates that the serving container does not have enough RAM to load the model and process the incoming request simultaneously. Online prediction requests can contain a batch of multiple instances. A large number of instances in a single request consumes significant memory. The most direct and immediate way to resolve this is to reduce the memory footprint of the request by sending a smaller batch of instances. This allows the request to be processed within the memory limits of the assigned machine type.

A. Switching to batch prediction is a different use case (asynchronous, high-throughput) and does not solve the OOM error for the required real-time online inference scenario.
C. Base64 encoding is for transmitting binary data in JSON but increases the data size by approximately 33%, which would likely worsen the memory issue, not resolve it.
D. A quota for the number of requests relates to throughput (requests per minute), not the memory allocated for processing a single request. Increasing this quota will not prevent an OOM error.

1. Google Cloud Vertex AI Documentation, "Get online predictions from a custom trained model": The documentation specifies that the request body contains a list of instances. It also notes a request size limit of 1.5 MB. An OOM error occurs when the processing of these instances exceeds the allocated memory of the serving node, directly linking request size (and thus, the number of instances) to memory consumption.

Source: Google Cloud. (n.d.). Get online predictions from a custom trained model. Vertex AI Documentation. Retrieved from https://cloud.google.com/vertex-ai/docs/predictions/get-online-predictions#requestbody (See section: "Request body" and "Limitations").

2. Google Cloud Vertex AI Documentation, "Deploy a model to an endpoint": This documentation details the selection of machine types for endpoints, which have specific vCPU and memory allocations. An OOM error is a direct consequence of the request's memory requirement exceeding the capacity of the chosen machine type. While changing the machine type is a long-term solution, reducing the request size is the immediate fix.

Source: Google Cloud. (n.d.). Deploy a model to an endpoint. Vertex AI Documentation. Retrieved from https://cloud.google.com/vertex-ai/docs/models/deploy-model-endpoint (See section: "Compute resources for prediction").

3. Stanford University, CS 329S: Machine Learning Systems Design, Lecture 11 "Model Serving": University courseware on MLOps explains that managing the batch size of inference requests is a critical parameter for controlling resource utilization, including memory. A larger batch size increases throughput but also memory requirements, and exceeding the available memory leads to errors like OOM.

Source: Stanford University. (2021). CS 329S: Machine Learning Systems Design, Lecture 11: Model Serving. (pp. 45-50). Retrieved from https://stanford-cs329s.github.io/slides/l11modelserving.pdf