Picture a thousand test cases sitting in Azure DevOps Test Plans, each one a paragraph of plain-English steps, and a QA team re-running the whole suite by hand every sprint. That was the starting point for a Microsoft engineering team that published its results, and it's a shape most teams with a mature manual suite will recognize: the coverage exists, it's just trapped as prose. The interesting part isn't that they automated it — it's how. They wired two Model Context Protocol (MCP) servers into a single agent so the test cases could be read on one side and Playwright code written out on the other, with a human steering in the middle.
I've rebuilt this flow, and it's genuinely replicable. Here's the walkthrough, plus an honest read on where it earns its keep and where you still need a person in the loop.
The two servers, and what each one is for
MCP is just a protocol that lets an agent call external tools through a uniform interface. The trick here is that the two servers do opposite jobs:
- The Azure DevOps MCP server (
@azure-devops/mcp, currently public preview) is the read side. It exposes your organization's work items, repos, wikis, and — the part that matters — test plans and test suites. The agent can pull a test case's title, steps, and expected results straight from ADO instead of you copy-pasting them.
- The Playwright MCP server (
@playwright/mcp) is the write side. It drives a real browser through Playwright's accessibility tree rather than pixels, so the agent navigates the app, inspects the structured snapshot, and generates test code grounded in what the page actually exposes — no vision model required.
One reads intent, the other produces working automation. The pipeline is the handoff between them.
Setup: one config file
Both run over stdio via npx, so a single .vscode/mcp.json at the project root wires up the whole pipeline for a GitHub Copilot agent in VS Code:
{
"inputs": [
{ "id": "ado_org", "type": "promptString", "description": "Azure DevOps organization" }
],
"servers": {
"ado": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@azure-devops/mcp", "${input:ado_org}"]
},
"playwright": {
"command": "npx",
"args": ["@playwright/mcp@latest"]
}
}
}
Hit Start on each server, switch the chat to Agent mode, open Select Tools, and enable the ADO and Playwright toolsets. The ADO server authenticates interactively by default — it pops a browser sign-in on first use, no PAT to manage. If the tool list feels noisy, the ADO server takes a -d flag to load only specific domains (like work-items or test-plans) instead of everything.
The pipeline: two prompts, not one
The single most useful finding from the source team is counterintuitive: splitting the work across two prompts produced more reliable code than one combined prompt. Ask for the fetch and the generation in a single breath and the agent tends to hallucinate steps or lose fidelity. Separate them and each stays grounded.
Prompt one — fetch the source of truth:
Get me the details of the test cases. Test Plan ID: 12345, Test Suite ID: 678.
The agent calls the Azure DevOps MCP server and returns the actual steps and expected results. You read them. This is your checkpoint — if the case is vague here, no amount of generation will save it.
Prompt two — generate the automation:
Convert the above test case steps to a Playwright script.
In practice you flesh prompt two out considerably: point at the project's page-object structure, the auth flow, an existing spec to match style, and your robustness conventions (locators over brittle selectors, explicit waits, no hard-coded sleeps). The more real context you feed it — existing tests, relevant files — the closer the first draft lands. The Playwright MCP server can take this further by opening the app, walking the flow live, and writing assertions against the snapshot it captures rather than guessing at selectors.
Then you validate locally, commit, and let Azure Pipelines run the generated specs. Loop the two prompts per case, or per suite, and the manual backlog drains steadily.
The result
The team reported automating hundreds of test cases in their own domain and, across the wider project, converting over a thousand — coverage that had previously only ever run by hand. That's the headline, but the mechanism is the point: plain-English descriptions became runnable specs without an engineer transcribing each one.
Where it helps, and where a human still gates
This is a drafting accelerator, not an autopilot. It shines when your test cases are already well-written and your app has a clean accessibility tree — the agent turns tidy prose into a solid first draft in seconds, which is exactly the tedious, mechanical work worth removing.
It stalls in three places, and all three are human gates:
- Garbage in, garbage out. A test case that reads "verify the report looks correct" generates a script that verifies nothing. You either sharpen the ADO descriptions up front or you rewrite the output. The pipeline makes your documentation debt visible, fast.
- No eyes. The Copilot agent can't interpret images or visual assertions — "compare against this screenshot" is beyond it. Chart rendering, pixel diffs, and visual regression still need Playwright's baseline screenshot assertions wired up by hand.
- Review is non-negotiable. Generated selectors drift, waits get missed, and a green run can still be asserting the wrong thing. Every draft goes through code review before it's trusted, the same as any human-written test.
The takeaway
If you have a large manual suite already documented in Azure DevOps, stand up both MCP servers in one mcp.json, and run the two-prompt loop — fetch, then convert — on your ten best-written test cases this week. You'll learn two things immediately: how much of the transcription genuinely disappears, and precisely which of your test cases were never specific enough to automate in the first place. Both are worth knowing, and the second one is the real deliverable.
Sources: From Manual Testing to AI-Generated Automation: Our Azure DevOps MCP + Playwright Success Story (Azure DevOps Blog) · microsoft/azure-devops-mcp (GitHub) · microsoft/playwright-mcp (GitHub)