AB-100 practice question 15 of 35
A contact center wants to tune a Copilot experience after a spike in incorrect next-best- action suggestions. You must demonstrate measurable…
Full set: 708 questionsThe best first decision to enable reliable, evidence-driven tuning under strict governance constraints is to establish an evaluation dataset with defined metrics. This dataset provides a standardized, repeatable benchmark (the "evidence") to measure the impact of any tuning change. It allows the team to quantify improvements, detect regressions, and make informed decisions before promoting changes through the dev/test/prod pipeline. This foundational step is a prerequisite for any systematic tuning effort and ensures that improvements are measurable and auditable, without requiring new data movement from the regulated environment.
A. Increase model temperature slightly: This is a specific tuning tactic, not a foundational decision. It should only be attempted after establishing a way to measure its effect.
B. Disable retrieval for speed: This is a drastic, untargeted action that would likely degrade accuracy and is irrelevant to the problem of incorrect suggestions.
C. Rely on CSAT only for tuning weekly: Customer satisfaction (CSAT) is a lagging, low-granularity metric and is insufficient for the repeatable, evidence-based tuning required.
1. Microsoft. (2024). Build and evaluate a question answering evaluation test set. Azure AI Services Documentation. Section: "Why evaluate?". This document explicitly states that an evaluation test set is crucial for measuring the accuracy of changes and providing confidence in the system's quality.
2. Microsoft. (2024). Evaluate and score a copilot's answers with a test set. Microsoft Copilot Studio Documentation. This guide details the process of creating and using test sets within the platform to systematically measure performance
which is the core of the correct answer.
3. Liang
P.
et al. (2022). Holistic Evaluation of Language Models. Stanford University Center for Research on Foundation Models (CRFM). Section 2: "Scenarios and Metrics". This foundational paper emphasizes the need for standardized scenarios (evaluation datasets) and metrics to enable rigorous
multi-faceted evaluation of AI models.
Keep practicing
Take the free AB-100 practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other AB-100 practice questions
- Question 2A global support team uses Dynamics 365 Customer Service and wants Copilot to summarize c…
- Question 3A manufacturing group is deploying an agent that drafts maintenance work orders from oper…
- Question 5A global manufacturer is rolling out a Microsoft 365 Copilot agent for supplier onboardin…
- Question 6A regulated insurer is modernizing a model-driven claims app and wants an AI-driven works…
- Question 7A global retailer is standardizing prompt guidance for agents used in Copilot Studio and …
- Question 8A global manufacturer is piloting an agent that drafts incident summaries, computes KPI v…
