Your team is running an AI inference workload on a Kubernetes cluster with multiple NVIDIA GPUs. You observe that some nodes with GPUs are…

Deploying a GPU-aware scheduler in Kubernetes (A) is the most effective strategy to balance GPU

workloads across a cluster. Kubernetes by default does not natively understand GPU resources

beyond basic resource requests and limits. A GPU-aware scheduler, such as the NVIDIA GPU Operator

with Kubernetes, enhances the orchestration by intelligently distributing workloads basedon GPU

availability, utilization, and specific requirements of the inference tasks. This ensures that

underutilized nodes are assigned work while preventing overloading of others, leading to consistent

performance.

Implementing GPU resource quotas(B) can limit GPU usage per pod, but it doesn’t dynamically

balance workloads across nodes—it only caps resource consumption, potentially leaving some GPUs

idle if quotas are too restrictive.

Using CPU-based autoscaling(C) focuses on CPU metrics and ignores GPU-specific utilization, making

it ineffective for GPU workload balancing in this scenario.

Reducing the number of GPU nodes(D) might exacerbate the issue by reducing overall capacity, not

addressing the imbalance.

The NVIDIA GPU Operator integrates with Kubernetes to provide GPU-aware scheduling, monitoring,

and management, making (A) the optimal solution.

Reference:NVIDIA GPU Operator documentation; Kubernetes integration with NVIDIA GPUs on

nvidia.com.