Open-source Playwright benchmark for evaluating AI web agents
Each tool is evaluated against our methodology using public docs, vendor demos, and hands-on testing.
What is Bananalyzer?
Bananalyzer is an open-source evaluation framework and benchmark dataset from Reworkd for testing AI web agents on structured information-retrieval tasks: detail pages, listing pages, and combinations of the two. It runs tasks against static MHTML snapshots of real websites using Playwright for browser automation, and users implement an AgentRunner interface to plug in their own agent. The tool installs via pip and runs through the bananalyze CLI. The stated goal is to unify existing web-agent datasets such as Mind2Web and WebArena into one format.
Our verdict
Useful for teams building web-browsing agents who want a ready-made, Playwright-based test harness under a permissive MIT license instead of building one from scratch. The catch: the last release and commit date to October 2024, with no activity since, and Reworkd's own product was reportedly discontinued in 2025. Treat this as a frozen snapshot, not an actively maintained benchmark.
Categories:
These are research benchmarks, not products you buy. They define fixed web tasks and score how reliably an agent or model completes them, which is the closest thing the field has to an objective measure of web-agent capability. They are listed here for evaluation, not procurement.
Open-source multimodal web agent with a 643-task benchmark
Free·Jul 2026·Benchmarks
Self-hosted, open-source benchmark for autonomous web agents
Free·Jul 2026·Benchmarks
Benchmark of 2,350 tasks across 137 real websites for web agents
Free·Jul 2026·Benchmarks
How Bananalyzer compares
Mind2WebMind2Web is one of the two datasets that Bananalyzer's README names as prior art, part of what it aims to unify into a common evaluation format.
WebArenaWebArena is the other benchmark named in Bananalyzer's README, one it aims to fold into its unified evaluation format for web agents.
WebVoyagerWebVoyager is a comparable benchmark for scoring AI agents on real-world web browsing tasks, in the same evaluation-framework category as Bananalyzer.
