Microsoft's Azure SRE Agent Handles Thousands of Incidents Autonomously, Freeing Engineers for Innovation

The scenario is familiar to site reliability engineers everywhere: a 3 a.m. alert forces someone to jump between monitoring dashboards and diagnostic tools, unsure whether the incident is genuine, whether they're the right person to handle it, or if they should escalate. By the time they piece together what happened, customers are already affected. Azure SRE Agent changes that dynamic. Rather than waiting for human intervention, the system analyzes telemetry, identifies root causes, and prepares fixes ready for approval and deployment.

Sanchit Mehta, a head engineer for Azure SRE Agent, explains the agent's approach: "[Azure SRE Agent] starts analyzing telemetry and correlates things like blast radius, deployment changes, recent changes, any recent rollouts, to try to tell the engineers, 'OK, this is what is causing it.'" The system increasingly generates pull requests for fixes as well.

The benefits extend beyond emergency response. At InEight, correlating telemetry across tens of thousands of Azure resources once consumed days or weeks. During the company's first incident using Azure SRE Agent, the system quickly pinpointed the affected product, traced the performance issue to its source, and recommended scaling Redis—a solution the DevOps team had been considering as a temporary workaround.

Scale and autonomy in production

Inside Microsoft, more than 3,000 service teams deploy Azure SRE Agent to investigate issues, perform root cause analysis, respond to incidents, fix code, enable automatic mitigation, support proactive detection, analyze data, and report at scale. The agent has already managed more than 1.8 million incidents within the company, with many resolved in minutes.

Microsoft's engineering teams also use Azure SRE Agent to develop and improve the service itself, deploying custom agents for code review, deployment, evaluation, and monitoring. This "agent-powered engineering" approach, as Mehta describes it, lets teams capitalize on advances in AI models. The system proactively identifies problems—such as quota issues affecting deployments—and automatically raises support tickets. In one instance, the agent identified the root cause of a change that broke synthetic tests the moment it reached the first region.

It said, 'OK, this was an upstream PyPI package that broke your dependency; you need to add tests for it; you should roll back immediately; here's how you should go fix this.'

Sanchit Mehta

Mehta notes that this kind of proactive monitoring is difficult to achieve with deterministic queries alone. "You need a level of intelligence to see when a large production payload is being deployed and if it has the potential to cause degradations."

For some internal teams, more than half of incidents are autonomously managed by the SRE agent without human intervention, according to Shamir Abdul Aziz, lead program manager for Azure SRE Agent. These are "safe" operations and mitigations: service restarts, scale-outs, rollbacks, or change order requests escalated by customers.

The humans did the governance, set up the guidelines, gave some coaching to the agent, and then it went into auto mode to complete the entire workflow

Shamir Abdul Aziz

When agents become necessary

Site reliability engineers face mounting repetitive toil, and the rise of coding agents threatens to compound that burden. Yet agentic operations have matured enough to help address this challenge, according to Vyom Nagrani, a head product manager for Azure SRE Agent.

As code gets written more and more by agents, it's going to take another agent to operate it. But why wait? If the agent can manage code which other agents write, why can't it manage code written by humans?

Vyom Nagrani

Nagrani adds that "the reasoning loop has become mature enough that now agents can automatically start figuring out a lot of these complex problems, especially when it comes to correlating across multiple data sources, which has always been the hardest thing for humans to do."

However, powerful models alone are insufficient. Homegrown automation lacks the production-grade governance, verification, evaluation, telemetry, and control that a platform provides. The field has progressed from prompt engineering to context engineering—grounding AI in infrastructure, code, and institutional knowledge—and now to harness engineering. "That is what allows you to run agents at scale, control them, and govern them," Abdul Aziz explains.

Verification, auditing, evaluation, and real telemetry transform agents from inscrutable black boxes into systems whose decisions can be traced and learned from. "That's why companies are willing to adopt it now," Abdul Aziz says. "Because when you try the same thing ten times, you're going to get the same output."

You don't just turn on the agent, give it full access, and ask it to solve everything.

Shamir Abdul Aziz

After two years of building enterprise-grade systems that can be trusted, audited, and validated, the next step for cloud-native SRE is agentic operations with autonomous capabilities. But adoption requires careful planning, Abdul Aziz cautions.

Building operational awareness

Azure SRE Agent is built for Azure but extends beyond it. The agent provides native access to Azure services including Azure Monitor, Application Insights, Log Analytics, and Azure Resource Graph. Connecting the agent to subscriptions, telemetry data, and source code supplies the operational context and institutional knowledge required to understand organizational workflows.

Beyond Azure, the agent integrates with engineering and operational tools through managed connectors for Azure DevOps and GitHub, plus MCP connectors enabling access to external knowledge sources such as Google Drive, Confluence, Cursor, Claude Code, and other third-party systems.

Storing knowledge in Markdown files within a repository, along with the skills and tools agents need to act on systems—including third-party and on-premises services—creates artifacts that agents can version, review, test, reuse, and update.

Custom agents can be created to dictate incident handling procedures: what to check and in what order, what to post, and how to format reports. These can be built using existing runbooks or by working through an incident with an agent and saving that skill. Using agents to improve agents accelerates the process of making Azure SRE Agent more effective over time, as agents learn from incidents and apply those lessons to future responses.

Governance, controls, and visibility

Governance encompasses identity, role-based access control (RBAC), and tool-access policies. These controls determine which actions are allowed, blocked, or subject to step-by-step approval, and whether an agent operates autonomously or with human review.

Hooks—based on prompts or deterministic commands—fire at different workflow stages and catch edge cases, such as allowing an agent to drop a SQL database index but never drop a table. Metrics demonstrate whether governance is working effectively. Live reports display time to mitigation, tool reliability, how often agents act autonomously, and cost per outcome.

InEight's results illustrate typical outcomes: an 80% reduction in both incident investigation time and build failure triage time, a 67% reduction in the effort needed to investigate bugs, and an 84% reduction in cost.

To achieve these results requires triggers that automatically launch agents rather than waiting for humans to open a chat window. Bind skills and custom agents to specific alert classes so they respond to incidents first. Start agents through pipelines, webhooks, or work items to automate delivery workflows. Schedule regular checks, reviews, and audits, and have agents automatically update their artifacts.

A maturity model for agent adoption

By reducing repetitive tasks and technical toil, Azure SRE Agent frees engineers to focus on more interesting and innovative projects. Organizations don't leap directly to having agents handle all operations. When agents receive context about infrastructure, they can begin with investigations.

If you give agents read access to your source code, your telemetry, your resources, the time to get to the root cause is reduced to minutes rather than hours or days. Every customer starts there.

Shamir Abdul Aziz

Once organizations are satisfied with investigation results, they can grant agents more permissions while still approving individual steps. "The fixing is easy once you understand the problem. It's usually changing your configuration, writing a piece of code, or restarting a service," Abdul Aziz explains.

As teams expand into other operational tasks, they should refine agent artifacts, metrics, and governance before granting additional autonomy. Examples include rolling back releases when regressions are detected, restarting services, dropping corrupt SQL indexes, or scaling out services.

For more complex issues, agents can deliver complete fixes ready for approval. The Azure SRE Agent managing the Azure SRE Agent product itself examines exceptions, errors, incidents, Teams conversations, emails, and GitHub issues each night and generates pull requests.

Avoid code review bottlenecks by having agents deploy, test, measure, and include outcomes in pull requests. Continuous evaluation builds a self-learning system that accurately follows existing workflows.

"The agent can self-improve because the agent learns constantly," Abdul Aziz notes. "You can configure scheduled tasks to identify which evaluation scores were low and automatically improve the custom agent, custom skills, and even your knowledge documents – because knowledge management is also a toil. The agent can automate all of that."

Getting started: Common pitfalls

Azure SRE Agent now offers a 30-day trial with no always-on charges. To maximize the trial period, organizations should learn from common mistakes:

• Agents are not magic. Enabling the agent does not eliminate the need for DevOps work. Avoid treating it as a chatbot or connecting it solely to observability systems. The agent requires context, tools to perform its job, and intentional triggers that signal when to act. Without these elements, it may expend effort on low-value work or generate outputs disconnected from the environment.
• Do not restrict yourself to out-of-the-box capabilities. Customize agent skills, tools, connections, and logic to match organizational workflows, and build custom agents for specific tasks.
• Do not use agents for jobs a single line of code can accomplish. Deploying them to explore deterministic, structured data for anomalies wastes tokens and floods the context window when the agent could write that code itself. "Orchestrate, don't calculate," as Nagrani puts it. If drowning in alerts, use automation to filter noise and send only alerts requiring intelligent analysis to agents.
• Do not replicate existing practices or organizational structures. The most effective agents possess a complete system picture, requiring all context even if it crosses team boundaries. This may mean crossing organizational lines, coordinating expertise and access, or rethinking how the organization operates.

"If agents have the right context, they minimize the toil and truly make operations less costly," notes Deepthi Chelupati, lead product manager. This approach enables faster movement, proactive operations, and gives engineers more time for innovation and less maintenance work to dread.