You recently developed a deep learning model using Keras, and now you are experimenting with different training strategies. First, you trained the model using a single GPU, but the training process was too slow. Next, you distributed the training across 4 GPUs using tf.distribute.MirroredStrategy (with no other changes), but you did not observe a decrease in training time. What should you do?

tf.distribute.MirroredStrategy implements synchronous data parallelism, where the global batch of data is split among the available GPUs. When you moved from one to four GPUs without changing the batch size, the per-GPU workload was reduced by a factor of four. This can lead to GPU underutilization, as the overhead of data transfer and gradient synchronization can dominate the small amount of computation time per step. To effectively leverage the multiple GPUs, you should increase the global batch size, typically scaling it by the number of devices. This ensures each GPU has a substantial amount of work, which is crucial for achieving a training speedup.

A. When using model.fit with a tf.distribute.Strategy, TensorFlow automatically handles dataset distribution across replicas. Manually calling experimentaldistributedataset is unnecessary and not the solution to the performance issue.
B. A custom training loop is not required to solve this problem. The standard model.fit API is fully integrated with tf.distribute and is capable of efficient distributed training when configured correctly.
C. Switching to different hardware (TPUs) is a premature and potentially costly step. The issue described is a common configuration problem with multi-GPU training, which should be resolved first.

1. TensorFlow Official Documentation, "Distributed training with Keras": This tutorial explicitly states the best practice for batch size. In the "Scale the batch size" section, it advises: "When training a model with tf.distribute.Strategy, it is a good practice to scale the batch size with the number of replicas to effectively utilize the extra computing power. This also requires you to adjust the learning rate accordingly." This directly supports increasing the batch size as the correct action.

(Source: TensorFlow, "Distributed training with Keras", section "Scale the batch size". Available at: https://www.tensorflow.org/tutorials/distribute/keras)

2. Google Cloud AI Platform Documentation, "Distributed training": The documentation on using distribution strategies highlights the relationship between workers and batch size. It explains that with MirroredStrategy, "each worker receives a portion of the input data," implying that if the global batch size is not increased, the per-worker portion shrinks, which can hurt performance.

(Source: Google Cloud, "Distributed training", section "Distribution strategies". Available at: https://cloud.google.com/ai-platform/training/docs/distributed-training)

3. Goyal, P., et al. (2017). "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." This seminal academic paper on large-scale distributed training establishes the principle of scaling the batch size with the number of workers. The authors demonstrate that for data parallelism, "When we use a minibatch of size kN, we perform k times fewer parameter updates to process the same number of images...". This foundational work validates that increasing batch size is the standard approach when increasing the number of processors.

(DOI: https://doi.org/10.48550/arXiv.1706.02677, Section 3, "Large Minibatch SGD")