While monitoring your model training’s GPU utilization, you discover that you have a native synchronous implementation. The training data is split into multiple files. You want to reduce the execution time of your input pipeline. What should you do?

The problem describes a classic input pipeline bottleneck where a synchronous implementation fails to keep the GPU supplied with data. Since the dataset is split into multiple files, the most effective solution is to parallelize the file reading and initial processing. The parallel interleave transformation reads from multiple files concurrently, interleaving their records. This breaks the serial, one-file-at-a-time bottleneck, significantly improving data throughput to the accelerator and reducing the overall training time by minimizing GPU idle time.

A. Increasing the CPU load is a symptom of a well-optimized pipeline, not a direct action one can take to fix a synchronous I/O bottleneck.
B. Caching is beneficial for subsequent epochs by storing preprocessed data in memory, but it does not solve the initial bottleneck of synchronously reading files in the first epoch.
C. Increasing network bandwidth is only relevant if the data is read from a remote location and the network is the confirmed bottleneck, which is not stated in the scenario.

1. TensorFlow Official Documentation, tf.data: Build TensorFlow input pipelines. In the section "Better performance with the tf.data API", under the subsection "Parallelizing data extraction", the documentation explicitly recommends this pattern: "To mitigate the effect of I/O latency, you can parallelize the data loading by using tf.data.Dataset.interleave. This transformation applies a user-defined function... to each element of the input dataset, and interleaves the results." This directly addresses reading from multiple files in parallel.

2. Google Cloud, AI Platform Documentation, "Optimize your input pipeline". This guide emphasizes that to prevent the CPU from becoming a bottleneck, it is crucial to "parallelize data preprocessing operations." It recommends using tf.data.Dataset.interleave to parallelize the reading of data from files, which directly supports the chosen answer. (Reference: Google Cloud > AI Platform > Training > Guides > Optimize your input pipeline).

3. Abadi, M., et al. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. The original TensorFlow whitepaper describes the system's dataflow graph model. Section 3.2, "Implementation," discusses how the graph's operations can be placed on different devices and executed in parallel. The tf.data API is a high-level implementation of these principles for efficient data input, where interleaving is a key pattern for parallel I/O. (Available at: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf, Section 3.2).