AB-100 practice question 35 of 35

You’re releasing an autonomous case-routing copilot that can open and close incidents across Dynamics 365 Customer Service and Sales. Production has…

Full set: 708 questions
You’re releasing an autonomous case-routing copilot that can open and close incidents across Dynamics 365 Customer Service and Sales. Production has strict separation of duties and you can’t use real customer transcripts outside the production region, but releases must happen weekly with no downtime. You need a repeatable test process with a quality gate that catches regressions in grounding and tool usage before promotion. Which process and metric gate best meets the constraints?
Answer options
A: Agent evaluation with regression gates Correct answer
B: Manual test chat with sign-off
C: Latency SLO gate only
D: Canary release using live refunds in prod only
Correct Answer:
A. Agent evaluation with regression gates
Explanation

The requirements call for a repeatable, automated quality gate to prevent regressions in grounding and tool usage before weekly production releases, without using real customer data in non-prod environments. 'Agent evaluation with regression gates' directly addresses this. This process involves running the agent against a standardized, curated set of test prompts (a "golden set") that cover critical grounding and tool-use scenarios. The results are automatically compared against a baseline, and the release is blocked (gated) if key metrics degrade. This provides a reliable, repeatable, and automated quality control mechanism that fits within a CI/CD pipeline and respects data boundaries.

Why Incorrect

B. Manual testing is not repeatable or scalable for weekly releases and is prone to human error, failing the "repeatable test process" requirement.
C. A latency SLO gate only measures performance. It does not catch functional regressions in grounding or tool usage accuracy, which is a key requirement.
D. Canary releasing in production uses live data, which violates the constraint of not using real customer data outside the production region for testing.

References

1. Microsoft Power Platform

"Application lifecycle management (ALM) with Microsoft Power Platform

" Solution concepts section. This document outlines the principles of using solutions to package and deploy components

which is a prerequisite for automated testing and promotion through environments.

2. Microsoft Copilot Studio Documentation

"Test your copilot." This page describes the built-in testing capabilities

which are the foundation for building more advanced

automated evaluation suites.

3. Microsoft Azure AI Studio Documentation

"Evaluate copilots and AI models." This documentation details the framework for systematic evaluation of AI agents against predefined datasets and metrics

which is the core of the "Agent evaluation" process.

Scroll to Top