Microsoft Open-Sources Scope, a Testing Framework for AI Coding Agents

While artificial intelligence coding agents can navigate SDK selection and tool invocation, they often stumble on tasks that human developers handle without difficulty. To address this gap, Microsoft developed Scope, which executes identical assignments across multiple agents and evaluates the quality of each outcome. The framework is now available under the MIT license as part of what Microsoft calls the Developer Agent Experience, or DevAx, initiative.

How does it work?

Scope accepts a task description, an optional starting codebase, and an agent configuration. The tool then directs existing agents such as GitHub Copilot and Claude Code through the specified task. Success criteria are structured as a dependency graph, where certain checks can block others. For instance, you might ask an agent to build a Node.js server and verify the presence of a GET /health route that returns JSON.

An LLM-based Judge service evaluates each execution against the defined criteria. Microsoft acknowledges that this scoring mechanism can produce errors, and the company recommends that critical findings still receive human validation. Every run generates live logs, output files, workspace snapshots, and records of tool invocations. A failed criterion can be investigated by tracing back through these artifacts. Agent configurations, including the model, skills, MCP servers, and extensions, are stored as versioned profiles.

You can replace one component, execute the same task again, and directly compare the results. Since outcomes fluctuate across runs, repeating tasks helps distinguish between persistent issues and isolated failures. Configurable personas allow you to adjust the perspective and communication style of the simulated user. Results can be examined through a web interface or accessed via CLI and REST API for integration into automation workflows.

Where would you use it?

• Documentation teams can identify instructions that agents consistently overlook, update the documentation, and retest the scenario.
• SDK maintainers can verify whether agents select their SDK initially and test how new sample code or added skills influence that decision.
• CLI and MCP server developers can examine traces to pinpoint where an agent gets stuck, revealing either unclear tool descriptions or ambiguous error messages.

What's the catch?

Scope does not offer a hosted service, meaning you must deploy and operate it yourself. There is no option to sign up and begin testing immediately.

Running Scope locally requires Docker with Compose v2, Node 22, pnpm, mkcert, and the GitHub CLI. Microsoft provides a local development guide to walk through the setup process. The default worker configuration requires an active GitHub Copilot subscription, and the Judge component needs credentials for a model provider. Organizations seeking a shared infrastructure can deploy Scope on Kubernetes using the provided Helm chart.

The source code is available on GitHub. Contributors should be prepared to sign the Microsoft Contributor License Agreement.