You’re releasing an autonomous case-routing copilot that can open and close incidents across Dynamics 365 Customer Service and Sales. Production has…

The requirements call for a repeatable, automated quality gate to prevent regressions in grounding and tool usage before weekly production releases, without using real customer data in non-prod environments. 'Agent evaluation with regression gates' directly addresses this. This process involves running the agent against a standardized, curated set of test prompts (a "golden set") that cover critical grounding and tool-use scenarios. The results are automatically compared against a baseline, and the release is blocked (gated) if key metrics degrade. This provides a reliable, repeatable, and automated quality control mechanism that fits within a CI/CD pipeline and respects data boundaries.

B. Manual testing is not repeatable or scalable for weekly releases and is prone to human error, failing the "repeatable test process" requirement.

C. A latency SLO gate only measures performance. It does not catch functional regressions in grounding or tool usage accuracy, which is a key requirement.

D. Canary releasing in production uses live data, which violates the constraint of not using real customer data outside the production region for testing.

1. Microsoft Power Platform, "Application lifecycle management (ALM) with Microsoft Power Platform," Solution concepts section. This document outlines the principles of using solutions to package and deploy components, which is a prerequisite for automated testing and promotion through environments.

2. Microsoft Copilot Studio Documentation, "Test your copilot." This page describes the built-in testing capabilities, which are the foundation for building more advanced, automated evaluation suites.

3. Microsoft Azure AI Studio Documentation, "Evaluate copilots and AI models." This documentation details the framework for systematic evaluation of AI agents against predefined datasets and metrics, which is the core of the "Agent evaluation" process.