During batch training of a neural network, you notice that there is an oscillation in the loss. How should you adjust your model to ensure that it converges?

Oscillation in the loss function during neural network training is a classic indicator that the learning rate is too high. The learning rate determines the step size the optimization algorithm takes to minimize the loss. If the steps are too large, the optimizer will repeatedly overshoot the optimal minimum point in the loss landscape, causing the loss to bounce back and forth, or oscillate. By decreasing the learning rate, the step size is reduced, allowing for a more stable and gradual descent towards the minimum. This prevents overshooting and helps the model to converge smoothly.

A. Increase the size of the training batch: This provides a more accurate gradient estimate, which can smooth the loss curve, but it does not fix the fundamental problem of overshooting caused by a large learning rate.
B. Decrease the size of the training batch: This introduces more variance (noise) into the gradient updates, which would likely exacerbate the loss oscillation rather than reduce it.
C. Increase the learning rate hyperparameter: This would make the optimization steps even larger, worsening the overshooting problem and likely causing the loss to diverge completely.

1. Stanford University CS231n Course Notes: In the section on training dynamics, it is explicitly stated and visually demonstrated that "A learning rate that is too high will cause the loss to diverge... you may see the loss oscillating." This directly supports decreasing the learning rate to fix oscillation.

Source: Stanford CS231n: Convolutional Neural Networks for Visual Recognition, Module 1, "Setting up the data and the model," subsection "Learning rate."

2. Google Cloud Documentation: The official documentation for Vertex AI hyperparameter tuning identifies the learning rate as a critical parameter that controls the training process. It explains that finding an optimal value is key to convergence, implicitly supporting the need to adjust it (in this case, decrease it) when observing undesirable behavior like oscillation.

Source: Google Cloud, Vertex AI Documentation, "Overview of hyperparameter tuning," section "Hyperparameters."

3. Academic Publication (Foundational Textbook): The "Deep Learning" book, a standard academic text, explains the challenges in neural network optimization. It notes that if the learning rate is too high, "the cost may actually increase after a parameter update," leading to oscillations or divergence.

Source: Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 8, "Optimization for Training Deep Models," Section 8.3.2, "Challenges in Neural Network Optimization," pp. 299-300.