You have just finished a prompt tuning experiment for a large language model (LLM) to optimize its output for generating customer support summaries…

The core issue is balancing inference speed (response time) with output quality. The decoding strategy directly controls this trade-off during text generation. Beam search explores multiple potential output sequences simultaneously to find a more optimal result, which is computationally expensive and slow. In contrast, greedy decoding simply selects the single most probable token at each step. This approach is significantly faster and less resource-intensive. While it may sometimes produce slightly lower-quality or less coherent summaries, it is the most effective parameter to adjust for a substantial improvement in response time with a potentially acceptable impact on quality.

A. Decrease the maximum token length: The problem states that a longer token length led to better quality. Decreasing it would directly and negatively impact the summary quality, which the goal is to avoid sacrificing significantly.
C. Increase the batch size: Batch size is related to throughput (processing multiple requests in parallel) during training or inference, not the latency (response time) for a single generation task.
D. Decrease the number of epochs: The number of epochs is a parameter for the model training process. The experiment is already finished, and this change would not affect the inference time of the tuned model.

1. Official Vendor Documentation:

IBM. (2024). watsonx.ai Documentation: Text generation parameters. In the section on decoding parameters

it is explained that the "Greedy" option "selects the token with the highest probability at each step in the decoding process

" contrasting with more complex methods. This highlights its role as a fundamental

speed-oriented decoding choice. (Reference: IBM watsonx.ai Docs > Foundation models > Text generation parameters > Decoding parameters).

2. University Courseware:

Manning

C.

& Jurafsky

D. (2023). CS224N: Natural Language Processing with Deep Learning

Lecture 8: Language Models and Recurrent Neural Networks. Stanford University. The lecture notes explain that greedy decoding is a fast but potentially suboptimal algorithm. It contrasts this with beam search

which "maintains a beam of the k most probable partial sequences at each step

" making it more computationally expensive but often yielding better results. This directly supports changing the decoding strategy to improve speed at the cost of some optimality. (Reference: Stanford CS224N Winter 2023 Lecture Slides

section on "Decoding from a Language Model").