An AI agent that triages CI build failures

A build goes red. Somewhere in a Playwright suite covering more than twenty feature areas, a test that passed yesterday has failed, and until somebody works out why, nobody knows whether the application is broken or the test is.

Answering that question, over and over, was one person's job. An agent does most of it now.

The problem

A long-running platform build for a large UK public-sector healthcare provider, through a consultancy I contract with. Azure underneath, MongoDB for storage, a Java and Spring backend, Azure DevOps running the pipelines and Jira holding the work.

The part that matters here is the test suite. Playwright end-to-end tests across more than twenty feature areas, running eight workers in parallel with two retries, on every pipeline run.

Eight parallel workers means failures arrive in clusters rather than one at a time. Two retries means anything that survives into a reported failure has already failed three times, so the easy flakes are filtered out before a human sees them and what is left is genuinely ambiguous.

One QA engineer held that surface. On a suite that size builds fail regularly, and every failure needed somebody to decide what it meant.

How this was handled before AI

The loop went like this, and it went like this every time.

Open the failing build in Azure DevOps. Scroll the log, which for a parallel run is eight workers' output interleaved, and find the test that actually failed rather than the ones cut short when the run gave up.

Read the error. Usually it is a timeout or an element that could not be found, which on its own tells you almost nothing. A missing element is what a genuine bug and a renamed CSS class both look like from the outside.

Go to the repository. Look at what has landed since the last green run. Is there a commit anywhere near this feature area, and did it touch the service behind the screen, the component the test drives, or the page object itself?

Try to reproduce it locally. Sometimes it reproduces. Often it does not, and now you know less than when you started.

Open the application and go and look. Navigate to the screen, open the inspector, find the element the test was reaching for, and check whether the selector in the page object still matches anything on the page.

Then write the ticket by hand. Summary, steps to reproduce, expected result, actual result, environment, a link back to the build. Then search Jira to check that one of the other seven workers has not already produced the same failure under a ticket somebody filed while you were reading.

None of that is difficult. All of it is reading, and all of it has to happen before anyone can say whether the software is broken. On a suite this size it is a queue that never empties.

What the AI actually changes

The agent runs the same loop. Claude Code sits in the middle, connected to the tools the team already had open, and when a build fails it opens the run, reads the log and pulls out the failing test.

Then it makes one decision. Every failure is classified as a code regression, a locator change, or a flake.

A code regression is a real behavioural change in the application. The test noticed that the software now does something different, and it was right to notice.

A locator change means the UI moved and the test's selector did not. Somebody renamed a data attribute or reorganised a form, the page object still points at the old thing, and the test now fails against an application that is working perfectly.

A flake is non-deterministic and would pass on another run. After two retries there are fewer of these than you would expect, which means most of what reaches a human is a real choice between the first two.

Git history is the discriminator. A commit touching the service behind the failing assertion points one way. A commit touching a component, a template or the page object points the other. Nothing relevant at all, on a test that has both passed and failed on the same code, points at a flake.

When the history is not enough, the agent opens a real authenticated browser session against the dev environment through Playwright and inspects the live DOM, checking whether the selector still matches anything on the current page. That is the step that makes a locator change decidable without a person reproducing anything.

Confirmed failures become tickets. The agent searches for an existing duplicate first, because eight parallel workers reliably produce several reports of one underlying problem, then writes the ticket in the project's established template: summary, steps to reproduce, expected and actual results, environment details, and a link back to the failing build.

The reported saving is roughly ten minutes per failing test.

It is easy to overstate what is new here. Fetching a log, parsing a stack trace and filling in a templated ticket are things a script could always have done, and plenty of teams have written that script. The classification and the DOM check are the parts that were not automatable before, because both are judgements made from messy evidence rather than rules you could write down in advance.

What actually wires up

MCP is the wiring that gives a model hands on tools you already run. Here it is enough to know that it is configuration.

Three connections did the work. Azure DevOps MCP, Microsoft's own server, to find the failing run and read its logs. As of September 2026 that one is public preview and supports Azure DevOps Services only, with no on-premises support. Atlassian Remote MCP Server, to search Jira for duplicates and create the ticket. Playwright MCP, also Microsoft's, for the browser session and the DOM check.

In Claude Code that is a file.

{
  "mcpServers": {
    "ado": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@azure-devops/mcp", "your-org"]
    },
    "atlassian": {
      "type": "http",
      "url": "https://mcp.atlassian.com/v2/mcp"
    },
    "playwright": {
      "type": "stdio",
      "command": "npx",
      "args": ["@playwright/mcp"]
    }
  }
}

One practical note before you copy it. Each of those servers exposes write actions as well as reads, and you choose which ones the agent gets. Creating a ticket and merging code are very different grants. Set that in your permissions settings rather than asking the model to behave in a prompt.

Config syntax checked September 2026 against the vendors' own documentation. This part of the ecosystem changes monthly, so read their repositories rather than trusting a blog post.

What a company would use, and what you can try for free

The version above is the paid path, and a team already inside Azure DevOps and Jira would build roughly that. The same shape runs on entirely open tools.

Alongside the triage work, that team also ran automated AI review on pull requests, a separate practice pointed at the same pipeline.

Trying it is much smaller than the project above. Install a runtime, add the hosted GitHub MCP server, and confirm it connects with its tools listed. Then ask it something you can check: classify the last five failed runs of a workflow as a code regression, a locator change or a flake, and say which commits you looked at. Read the tool calls before you read the answer, because the question is whether it went and looked or wrote something plausible from the workflow name.

Sources

• Public case study for this engagement: CI/CD failure triage with Claude Code and MCP. The stack, the suite configuration, the three-way classification, the ticket workflow and the roughly ten minutes saved per failing test.
• Model Context Protocol specification, version 2025-11-25, and the official registry for finding servers with verified publishers.
• The official servers used above: Azure DevOps MCP, Atlassian Remote MCP Server, Playwright MCP and GitHub MCP Server, including the preview and on-premises caveats.
• Claude Code MCP configuration and its permission settings, for the config format and the allow and deny rules.
• Open runtimes and tooling: Goose, OpenHands, Langfuse and Ollama.