During a multi-day NeMo burn-in, intermittent "GPU fell off bus" errors occur. Which diagnostic approach isolates hardware faults?

The "GPU fell off the bus" error indicates a severe hardware communication failure on the PCIe bus, which is almost always a hardware fault. The most effective method to isolate such faults is to run comprehensive hardware diagnostics. NVIDIA Data Center GPU Manager (DCGM) provides the dcgmi diag tool for this purpose. Running a medium (-r 3) or long (-r 4) duration diagnostic performs intensive, targeted stress tests on GPU components, including memory, PCIe interface, and processing cores. This process is designed to reproduce and pinpoint intermittent hardware issues that may not appear under normal workloads, thereby isolating the faulty component.

A) Reinstalling drivers is a solution for software corruption, which is an unlikely cause for a "fell off the bus" hardware-level PCIe error.

C) NeMo application logs operate at a much higher software layer; the error originates from the kernel, indicating a problem far below the application.

D) While overheating can cause instability, this specific error points more directly to a PCIe link or power fault, which DCGM diagnostics also test for.

1. NVIDIA Data Center GPU Manager (DCGM) User Guide. (Version 3.1.8). Section 5.1 "Diagnostic". The documentation describes the diagnostic as a tool to "run a comprehensive set of tests to check for hardware and software problems." It details the different test levels, with Level 3 including "PCIe, memory, and SM stress tests" designed to find hardware faults.

2. NVIDIA DGX A100 User Guide. (DU-10228-001v05). Chapter 8 "Troubleshooting". This guide recommends running DCGM diagnostics (dcgmi diag -r 3) as a primary step for troubleshooting GPU hardware issues, including PCIe errors.