Playwright MCP vs CLI? Benchmarking for performance & effectiveness

I benchmarked an agent using Playwright MCP vs Playwright CLI vs agent-browser. The CLIs were much more context efficient, but they were–surprisingly–also much slower! I dug into why that might be and the answer came down to tool calls, which turned out to be driven directly from the interface designs of each tool.

The MCP used up a lot of the context window by returning all state information on every run. The CLI and agent-browser both returned much less information, leaving it up to the agent to directly query the information it needed. That meant less of the context window was used, but more tool calls were required and tool calls were the most direct corollary to timing.

The biggest surprises were that context efficient != faster, and it also didn’t mean cheaper.