You have deployed an AI training job on a GPU cluster, but the training time has not decreased as expected after adding more GPUs. Upon further…

The data preprocessing being bottlenecked by the CPU is the most likely cause. High CPU utilization

and low GPU utilization suggest the GPUs are idle, waiting for data, a common issue when

preprocessing (e.g., data loading) is CPU-bound. NVIDIA recommends GPU-accelerated

preprocessing (e.g., DALI) to mitigate this. Option A (model incompatibility) would show errors, not

low utilization. Option B (connection issues) would disrupt communication, not CPU load. Option C

(software version) is less likely without specific errors. NVIDIA’s performance guides highlight

preprocessing bottlenecks.

Reference: NVIDIA Deep Learning Performance (developer.nvidia.com), NVIDIA DALI

(developer.nvidia.com).