A contact center wants to tune a Copilot experience after a spike in incorrect next-best- action suggestions. You must demonstrate measurable…

The best first decision to enable reliable, evidence-driven tuning under strict governance constraints is to establish an evaluation dataset with defined metrics. This dataset provides a standardized, repeatable benchmark (the "evidence") to measure the impact of any tuning change. It allows the team to quantify improvements, detect regressions, and make informed decisions before promoting changes through the dev/test/prod pipeline. This foundational step is a prerequisite for any systematic tuning effort and ensures that improvements are measurable and auditable, without requiring new data movement from the regulated environment.

A. Increase model temperature slightly: This is a specific tuning tactic, not a foundational decision. It should only be attempted after establishing a way to measure its effect.

B. Disable retrieval for speed: This is a drastic, untargeted action that would likely degrade accuracy and is irrelevant to the problem of incorrect suggestions.

C. Rely on CSAT only for tuning weekly: Customer satisfaction (CSAT) is a lagging, low-granularity metric and is insufficient for the repeatable, evidence-based tuning required.

1. Microsoft. (2024). Build and evaluate a question answering evaluation test set. Azure AI Services Documentation. Section: "Why evaluate?". This document explicitly states that an evaluation test set is crucial for measuring the accuracy of changes and providing confidence in the system's quality.

2. Microsoft. (2024). Evaluate and score a copilot's answers with a test set. Microsoft Copilot Studio Documentation. This guide details the process of creating and using test sets within the platform to systematically measure performance, which is the core of the correct answer.

3. Liang, P., et al. (2022). Holistic Evaluation of Language Models. Stanford University Center for Research on Foundation Models (CRFM). Section 2: "Scenarios and Metrics". This foundational paper emphasizes the need for standardized scenarios (evaluation datasets) and metrics to enable rigorous, multi-faceted evaluation of AI models.