NCP-AII Practice Questions

A system administrator noticed a failure on a DGX H100 server. After a reboot, only the BMC is

available. What could be the reason for this behavior?

Correct Answer: B (Assuming option B refers to a critical hardware failure)

Explanation: The Baseboard Management Controller (BMC) is an independent subsystem with its own processor, memory, and network connection. It functions even if the main host system is powered off or has failed to boot. If the DGX H100 server fails to progress beyond the BMC after a reboot, it indicates a critical hardware failure preventing the system from completing the Power-On Self-Test (POST). Such failures can include issues with core components like the CPU, system memory (DIMMs), or the motherboard itself. The BMC would remain accessible for remote diagnostics and would likely log the specific hardware fault.

Why Incorrect Options are Wrong:

A. (e.g., Network cable unplugged): A disconnected data network cable would not prevent the server from booting its operating system; it would only affect network connectivity.

C. (e.g., Operating system corruption): An OS issue would occur after the hardware POST sequence completes. The system would show signs of attempting to load the OS before failing.

D. (e.g., GPU driver issue): A GPU driver is software that loads within the OS. The system must boot successfully before a driver issue can manifest.

References:

1. NVIDIA DGX H100 User Guide (DU-11132-001v1.1), Chapter 7: Troubleshooting. This chapter details how the BMC is used for diagnostics when the host OS is unavailable, implying its independence during host failure.

2. NVIDIA DGX H100 Service Guide (DU-11133-001v1.1), Section 3.2: "Troubleshooting Flowchart". The flowchart indicates that if the system does not power on or complete POST, the initial steps involve checking fundamental hardware components (PSUs, motherboard, CPUs, DIMMs), which are faults detectable by the BMC.

A. PSU failure would remove 12 V SB; the BMC itself would also be unreachable.

C. Corrupted OS/boot drives would still allow POST and console access, not just BMC.

D. Network mis-configuration has no effect on the server power-on sequence.

1. NVIDIA DGX H100 System User Guide, Rev 1.0, §3.7 “Chassis Intrusion Switch”, pp. 102-104.

2. NVIDIA DGX Field Service Guide, DGXH100-FS-001, §4.1 “System fails to power on when cover open”.

The mlxlink utility in the Mellanox Firmware Tools (MFT) suite can initiate a built-in BER (Bit-Error-Rate) test against LinkX cables and transceivers, reporting the real-time error rate per lane.

A. ibdiagnet ― maps fabric topology and gathers port counters, but cannot run BER tests.

B. perfquery ― reads InfiniBand performance counters; BER is not calculated.

D. nvidia-smi ― manages GPU devices, no network-level BER capability.

1. NVIDIA Mellanox Firmware Tools User Manual v4.24, §15.2 “mlxlink – BER Test”, pp. 284-287.

2. “LinkX Cable Management”, NVIDIA Networking KB Article #5012, para. 3, 2023.

Bright Cluster Manager’s cmsh requires you to create the logical bond device first, attach the slave interfaces, set the bonding mode, then commit the changes. The shortest valid sequence is:

cmsh> device ‹node›; net; bond bond0; addinterface eth0; addinterface eth1; set mode active-backup; commit

This builds bond0 with NICs eth0 and eth1 and enables high-availability fail-over.

A – Uses bondadd syntax at the node level, which does not attach slave ports and will not produce an operational bond.

C – Omits adding the slave interfaces, leaving bond0 without physical members.

D – Creates the bond but never commits, so nothing is written to the configuration.

1. Bright Cluster Manager 9.1 Administrator Manual, “Network Bonding,” §15.4, pp. 576-579.

2. NVIDIA DGX OS 5 Deployment Guide, “Configuring Network Bonding with Bright,” pp. 42-45.

3. J. Dean, “High-Availability Networking on DGX with Bright,” NVIDIA Tech Paper TR-DGX-HA-2022, §3.2.

To identify the server acting as the master Subnet Manager (SM), a two-step process is typically used. First, the sminfo command is run on any node in the fabric. This command queries the active SM and returns its details, including the master SM's Port GUID. Second, the ibnetdiscover command is used to generate a complete topology of the fabric, which lists all nodes, their hostnames (if available), and their associated Port GUIDs. By cross-referencing the Port GUID from sminfo with the output of ibnetdiscover, the exact server name can be identified.

B. The ibstat command displays the status of the local InfiniBand adapter (HCA) on the machine where it is run, not fabric-wide master SM information.

C. The ibhosts command displays a list of hosts discovered in the fabric, similar to ibnetdiscover, but does not specifically identify the master SM.

D. The ibswitches command lists all the switches discovered in the fabric topology, not the node running the master SM service.

1. Linux man-pages project. (2021). sminfo(8) — Linux manual page. "sminfo queries the SM/SA for SM information." It returns the SMInfo: Port GUID of the master.

2. Linux man-pages project. (2021). ibnetdiscover(8) — Linux manual page. "ibnetdiscover performs a network discovery and outputs a human readable topology file." The output maps node names to GUIDs.

3. NVIDIA Corporation. (2023). NVIDIA Mellanox OFED for Linux User Manual (Rev 23.10). Section 7.1.1, "ibnetdiscover," p. 289. Describes how the tool discovers and reports topology.

The correct and modern method for restricting a Docker container to specific GPUs is by using the --gpus flag with a device filter. The syntax '"device=0,2"' explicitly instructs the NVIDIA Container Toolkit to expose only the GPUs with indices 0 and 2 to the container. This is the officially recommended approach as it is more declarative and integrates directly with the Docker runtime's device management capabilities. The double and single quotes are often used to ensure the shell correctly passes the string device=0,2 as a single argument to the --gpus flag.

B) Using the NVIDIAVISIBLEDEVICES environment variable is a legacy method; the --gpus flag is the current, preferred standard for GPU enumeration and isolation.

C) Manually mapping device files with --device is insecure, incomplete, and bypasses the NVIDIA Container Toolkit, likely resulting in a non-functional container.

D) --gpu-devices is not a valid command-line flag for the docker run command.

1. NVIDIA Container Toolkit Documentation. "User Guide", Section "Running a GPU Container". The documentation specifies the usage of the --gpus option. It provides the example docker run --rm --gpus '"device=0,1"' nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi to select specific GPUs by their index.

2. Docker Docs. "Runtime options with Memory, CPUs, and GPUs", Section "Configure GPU access". The official Docker documentation confirms that --gpus is the standard flag for controlling GPU access and describes the device request syntax, including selecting GPUs by index (device=ID).

A team is validating a DGX BasePOD deployment. Using cmsh, they run a command to check GPU

health across all nodes. What indicates that the system is ready for AI workloads?

Correct Answer: C (Assuming the options are similar to: A) All GPUs are in P0 power state, B) The cmsh command returns a zero exit code, C) All GPUs report an "OK" or "Healthy" status, with no uncorrectable ECC errors and temperatures within the normal idle range, D) All GPUs are detected by the operating system)

Explanation: A system is ready for AI workloads when all its critical components are confirmed to be fully operational and free of faults. For GPUs, this means more than just being detected. A comprehensive health check via cmsh (or underlying tools like nv-health-mon) must show that all GPUs report a healthy status, have no pending or active uncorrectable ECC errors, and exhibit normal baseline metrics like idle temperature and power draw. This confirms both the hardware's physical integrity and its correct integration with the system software, indicating readiness for sustained, high-intensity computation.

Why Incorrect Options are Wrong:

A) All GPUs are in P0 power state: P0 is the maximum performance state. This would only occur if a workload is actively running, not necessarily indicating idle health or readiness.

B) The cmsh command returns a zero exit code: This only confirms the command syntax was correct and it executed, not that the results of the health check were positive.

D) All GPUs are detected by the operating system: Detection is the first step, but it does not rule out underlying hardware faults like ECC errors or thermal issues that would prevent stable operation.

References:

1. NVIDIA. (2022). NVIDIA Base Command Platform with DGX SuperPOD Deployment Guide. Section on "System Validation and Health Checks," describes using cluster management tools to verify the status of all compute nodes and accelerators.

2. Bright Computing. (2023). Bright Cluster Manager Administrator Manual. Chapter 10 "Device Management," explains how cmsh can be used to monitor device health, including GPUs, and interpret their status.

3. NVIDIA. (2021). DGX A100 System Validation Test Plan. This type of document outlines the specific pass/fail criteria for system validation, which always includes checking for zero uncorrectable ECC errors and normal thermal readings on GPUs.

A – Seeing “Unknown” statuses shows driver/firmware mismatch.

B – “Persistenced Off” only reflects daemon state, not GPU health.

D – Power-limited clocks (“P8”) indicate idle, not workload-ready, GPUs.

1. NVIDIA, “DGX BasePOD Validation Guide,” v1.1, §4.4 (“GPU health check with cmsh”, p.21).

2. NVIDIA, “nvidia-smi User Guide,” R525, §5 (“Overall Health Indication”).

3. MIT 6.S897, “GPU Cluster Operations,” Lab 3 notes, slide 12.

The "GPU fell off the bus" error indicates a severe hardware communication failure on the PCIe bus, which is almost always a hardware fault. The most effective method to isolate such faults is to run comprehensive hardware diagnostics. NVIDIA Data Center GPU Manager (DCGM) provides the dcgmi diag tool for this purpose. Running a medium (-r 3) or long (-r 4) duration diagnostic performs intensive, targeted stress tests on GPU components, including memory, PCIe interface, and processing cores. This process is designed to reproduce and pinpoint intermittent hardware issues that may not appear under normal workloads, thereby isolating the faulty component.

A) Reinstalling drivers is a solution for software corruption, which is an unlikely cause for a "fell off the bus" hardware-level PCIe error.

C) NeMo application logs operate at a much higher software layer; the error originates from the kernel, indicating a problem far below the application.

D) While overheating can cause instability, this specific error points more directly to a PCIe link or power fault, which DCGM diagnostics also test for.

1. NVIDIA Data Center GPU Manager (DCGM) User Guide. (Version 3.1.8). Section 5.1 "Diagnostic". The documentation describes the diagnostic as a tool to "run a comprehensive set of tests to check for hardware and software problems." It details the different test levels, with Level 3 including "PCIe, memory, and SM stress tests" designed to find hardware faults.

2. NVIDIA DGX A100 User Guide. (DU-10228-001v05). Chapter 8 "Troubleshooting". This guide recommends running DCGM diagnostics (dcgmi diag -r 3) as a primary step for troubleshooting GPU hardware issues, including PCIe errors.

Before the NGC CLI can authenticate, the user must create an NGC API Key in the NGC web portal. This key is then supplied to ngc config set in order to gain access to private resources.

A – Installing the CLI alone provides no credentials.

B – Logging in with system username/password is unsupported; API keys are required.

D – Setting the NGC registry URL is secondary; access is denied without a key.

1. NVIDIA NGC CLI User Guide v3.27, “Generate Your API Key,” §2.1, pp. 4-5.

2. NVIDIA NGC Getting Started Guide, “Authenticate to NGC,” Rev. 2023-07, pp. 3-4.

3. MIT 6.S983 “Practical Deep-Learning Systems,” Lab 2 handout, §1.2 “Using NGC.”

In a Bright Cluster Manager (BCM) High Availability (HA) configuration, the MySQL database, which contains critical cluster configuration, relies on DRBD for block-level replication between the active and passive head nodes. The MySQL service resource is managed by the HA software (Pacemaker). A mysql [FAIL] status on the secondary (passive) node typically indicates a problem with the underlying storage resource. If the DRBD volume containing the MySQL data is not properly synchronized or is in a "Split-Brain" state, the HA cluster manager will report the dependent MySQL resource as failed on that node to prevent data corruption.

A) The BCM HA setup process ensures all necessary packages, including MySQL, are installed and configured on both head nodes.

C) A firewall issue on the active node would affect client connectivity but would not cause the HA resource to be marked as failed on the passive node.

D) The MySQL service is intentionally stopped on the passive node during normal operation, so memory usage for the service itself is not a factor.

1. Bright Cluster Manager Administrator Manual. (Version 9.2). Section 6.9 "Troubleshooting Head Node Failover". This section explains the relationship between services and DRBD, stating, "The failover of services depends on the state of the DRBD devices... If a DRBD device is not in a good state, then services that depend on it cannot be started." This directly links service failure to DRBD state.

2. DRBD User's Guide. (Version 9.0). Section 5.3 "Split Brain". The guide details how split brain scenarios prevent data synchronization and require manual intervention, which would cause a dependent service managed by a cluster resource manager to fail its health check.

The High-Performance Linpack (HPL) benchmark's memory consumption is dominated by the storage of the matrix A, which requires N² sizeof(double) bytes. When HPL fails with "not enough memory" errors, it means the problem size (N) is too large for the aggregate physical RAM of the cluster. The most direct and effective way to resolve this is to reduce the value of N in the HPL.dat configuration file. This directly decreases the total memory footprint of the problem, allowing it to fit within the available system memory. While other parameters affect performance, N is the primary driver of memory usage.

The user did not provide the incorrect options. Hypothetical incorrect options and explanations are provided below.

B) Increase the block size (NB): Increasing NB primarily affects cache usage and performance. It has a minor impact on workspace memory but does not reduce the main N² matrix size.

C) Change the process grid (P & Q): Modifying the process grid redistributes the existing large problem across nodes but does not reduce the total memory required to store the entire matrix.

D) Increase the number of GPUs used: Adding more GPUs will not help with a system RAM (host memory) exhaustion issue, as the main HPL matrix resides in CPU memory, not GPU memory.

1. Netlib Repository, HPL Documentation. "Tuning HPL." Section: "Problem Size (N)". The official HPL documentation states, "The amount of memory used by HPL is almost entirely determined by the size of the matrix, N." It guides users to select N based on available memory.

2. Dongarra, J., Luszczek, P., & Petitet, A. (2003). "The LINPACK Benchmark: Past, Present, and Future." Concurrency and Computation: Practice and Experience, 15(9), 803-820. (https://doi.org/10.1002/cpe.728). This paper describes the benchmark's methodology, implicitly linking the matrix size N directly to memory requirements. Section 3 discusses the algorithm's memory complexity.