Can AI Browser Agents Actually Complete Tasks? A Real-World Tool Comparison

AI browser agents work. They also fail more often than the demos suggest, and the gap between those two facts is the whole story. On WebArena, 812 tasks run against self-hosted replicas of shopping, forum, code-hosting and content-management sites, the best tracked system scores 74.3%. On Online-Mind2Web, 300 tasks run against 136 live websites and graded by human annotators, OpenAI Operator scores 61% and Claude Computer Use 3.7 scores 56.3%, while every other agent tested lands near 30%. Same capability, same year, wildly different numbers.

That difference changes which claims you can use to plan a deployment. A benchmark score needs context: the tasks, the website environment and the grading rules. Price and terms of service can rule out a tool even when its score looks good.

The short answer: reliable on fixed sites, unreliable on the live web

An AI browser agent can complete multi-step web tasks reliably when the environment is controlled and a human can intervene. It cannot yet complete them unsupervised on the open web at anything close to the rates vendors imply in product pages.

Three facts frame the decision. First, the reliability ceiling is set by task length, not by model quality: the Online-Mind2Web team found average success fell 31.6% moving from easy to medium tasks, then another 15.4% from medium to hard. Second, cost scales with steps, and failed steps are expensive: failed tasks consumed nearly twice as many steps as successful ones in three of the six agents tested. Third, the legal and technical perimeter moved under everyone’s feet on 15 September 2026, when Cloudflare began blocking Agent-category bots by default on ad-supported pages for newly onboarded domains.

Where the benchmark numbers come from

Prices and plan limits reflect vendor pages checked on 16 October 2026. Benchmark scores are attributed to their papers, project pages or leaderboards, with the task count, environment and grading method stated. No agent tasks were run for this comparison.

Steel.dev and BenchLM aggregate results. Steel notes that community-tracker rows are not always independently verified, so a score without an underlying source is marked as single-sourced. Perplexity Comet, Manus and Genspark appear without scores because no reproducible primary-source result is cited here.

The reliability curve: four benchmarks, four different pictures

The benchmarks disagree about how good these agents are. That disagreement is informative, not confusing, because each one isolates a different variable.

Benchmark	Tasks and environment	Grading	Best verified result	Human baseline	Source
WebArena	812 tasks, self-hosted replicas of 4 site types	Programmatic state checks and answer matching	WebTactix on DeepSeek v3.2, 74.3%	78.24%	Zhou et al., arXiv:2307.13854; Steel.dev leaderboard, updated 11 Oct 2026
VisualWebArena	910 tasks, same replicas plus a classifieds site, visual comprehension required	Execution-based tests on annotated screens	GPT-4o with Set-of-Marks, 19.78%	88.70%	Koh et al., ACL 2024, arXiv:2401.13649; jykoh.com/vwa
WebVoyager	643 tasks, 15 live public websites	GPT-4V automatic judge, 85.3% agreement with humans	browser-control on Claude Fable 5, 99.19%	Not reported in the paper	He et al., ACL 2024, arXiv:2401.13919; Steel.dev leaderboard
Online-Mind2Web	300 tasks, 136 live websites	Human annotation, reference standard	OpenAI Operator, 61%	Not reported	Xue et al., COLM 2025, arXiv:2504.01382v4
Mind2Web-Live (WebCanvas)	542 tasks, 2,439 intermediate evaluation states, live pages	Key-node progress scoring	23.1% task success, 48.8% task completion	Not reported	Pan et al., arXiv:2406.12373
OSWorld	369 tasks, real desktop environments	Execution-based state validators	Qwen3.8-Max, 86.1%, self-reported	72.36% in the original paper	Steel.dev OSWorld leaderboard; OSWorld paper

Read the WebVoyager column last and you will see the problem in one number. A 99.19% success rate sounds like solved automation. It comes from an automatic GPT-4V judge that the original WebVoyager authors themselves said agrees with human judgment only 85.3% of the time. The Online-Mind2Web team tested that judge directly and found its agreement rate with humans degrades sharply as trajectories get longer, because the token load of accumulating screenshots causes failures, and it separately attributed inflated historical scores partly to hallucinated final responses being read as evidence of success.

“Frontier web agents show a drastic drop in success rate when evaluated on Online-Mind2Web (human evaluation) compared with those reported on WebVoyager. Surprisingly, many recent agents, except for Claude Computer Use 3.7 and Operator, do not outperform the simple SeeAct agent released in early 2024.”: Xue et al., An Illusion of Progress? Assessing the Current State of Web Agents, COLM 2025

Why the same agent scores 74% on one benchmark and 30% on another

Three structural reasons, each documented rather than guessed.

The environment does not move. WebArena runs self-hosted site replicas. That is the point: it gives you reproducibility. It also means no layout drift, no session expiry, no CAPTCHA, no A/B test, no login wall mid-run. BenchLM’s own methodology page makes the admission directly, noting that fixed self-hosted sites do not test layout drift, anti-bot systems, production login flows, real-world identity and permission boundaries, safety controls, latency or cost.

The grader rewards the wrong thing. WebArena checks end state programmatically, so several action paths can pass. That is rigorous for what it measures and silent about everything else. WebVoyager’s GPT-4V judge is the opposite problem: it is cheap and scalable and it occasionally marks a task complete because the agent hallucinated a plausible answer.

Difficulty is doing the work. Online-Mind2Web grades tasks by the number of steps a human needed. Claude Computer Use 3.7 and Operator scored 90.4% and 83.1% on easy tasks. Those same systems collapse on medium and hard ones. If your production task is a five-step form fill, you are buying from the easy tier. If it is a fifteen-step procurement workflow across three portals, you are buying from the tier where nobody clears 30%.

One more finding deserves attention because it is counter-intuitive. OpenAI Operator, the best performer on Online-Mind2Web, was also the slowest. It explored heavily, taking 2.6 times the human reference number of steps and up to 44 minutes on challenging tasks. Other agents were more efficient, at roughly the human step count, but got stuck at dead ends. Reliability and efficiency came from opposite strategies, and you have to pick one deliberately.

Which AI browser agents you can actually buy in October 2026

The category churned hard this year. ChatGPT Atlas, OpenAI’s Mac browser, was retired and no longer receives updates, according to OpenAI’s own help centre, after under a year. TechCrunch reported on 9 July 2026 that the browser’s capabilities were redistributed into the ChatGPT desktop app, a Chrome extension and a separate cloud browser. ChatGPT agent mode itself remains live: OpenAI’s help centre, read on 16 October 2026, documents it on paid plans.

Product	Type	Entry price, checked 16 Oct 2026	Published task success rate	Evidence quality
ChatGPT agent mode	Consumer agent inside ChatGPT	Plus $20/mo, 40 messages/mo; Pro 400 msgs/mo	BrowseComp 68.9%, vendor-reported	Vendor self-report, July 2025
Claude in Chrome	Browser extension	Bundled from Pro at $20/mo or $17/mo annual; Max from $100/mo	None published for the product	Vendor red-teaming published, no accuracy benchmark
Anthropic browser use tool	API toolset	Claude Sonnet 5.5 at $2/$10 per MTok	None for the tool itself	Claude Sonnet 5 scores 81.2% on OSWorld, self-reported
Browser Use (open source)	Python and TypeScript library, MIT	Library free; cloud browsers from $0.02/hour	89.1% on WebVoyager, Dec 2024, GPT-4o	Vendor-reported, Dec 2024, old harness
Stagehand (Browserbase)	TypeScript SDK over Playwright primitives	Open source; Browserbase Free $0, Developer $20/mo, Startup $99/mo	None published	Vendor claims performance, no published benchmark
Skyvern	Open-source agent, AGPL-3.0	Free 5,000 credits; Hobby $29/mo; Pro $149/mo	85.85% on WebVoyager, self-reported	Vendor-reported, Jan 2025
Amazon Bedrock AgentCore Browser	Managed browser runtime	Per-second CPU and memory billing; Web Search $7/1,000 queries	None published	AWS documented; no benchmark
Steel.dev	Browser infrastructure API	Launch $30 credit, $0.10/browser-hour; Scale $100/mo, $0.08/hour	Publishes the leaderboards, not an agent	Infrastructure vendor
Hyperbrowser	Browser infrastructure plus per-step agent billing	Free 5,000 credits; Startup $30/mo; Scale $100/mo	HyperAgent $0.02/step, Computer Use $0.02/step	No benchmark
Airtop	Managed agents	Free $0, Starter $29/mo, Professional $189/mo, Enterprise $558/mo	None published	Vendor claims self-healing agents
Playwright MCP	Microsoft MCP server over Playwright	Free, MIT	None published	Widely used; Microsoft publishes no accuracy claim

Two rows deserve comment. Playwright MCP is the one I would reach for first for a repeatable internal workflow, because it drives the accessibility tree rather than pixels, needs no vision model, and applies tools deterministically. Microsoft’s own README now steers coding agents toward a CLI-plus-skills path instead, explicitly because MCP loads large tool schemas and verbose accessibility trees into context.

And note the last column. Most products publish no task success rate at all. Anchor Browser, for example, advertises being “12X faster,” using “80X less tokens” and being “2–3x less error prone compared to the best performing browser agents.” Those are vendor marketing claims with no published methodology, without an independent benchmark cited here.

What it costs, and why a browser task costs 10 to 100 times an API call

Here is my own arithmetic, built from published per-token rates and Anthropic’s published token-overhead figures. It is a calculation, not a measurement, and I am labelling it as such.

Assumptions: 20 turns, one 1280×800 screenshot per turn, roughly 400 tokens of text per tool result, roughly 350 output tokens per turn. Anthropic’s vision documentation states that images cost ceiling of width ÷ 28 × ceiling of height ÷ 28 visual tokens, which for 1280×800 is 46 × 29 = 1,334 tokens, and that long agent loops accumulate roughly 1,000 to 1,800 input tokens per screenshot.

The catch that makes this expensive is that a multi-turn conversation re-sends the whole prefix every turn. Turn 20 contains 20 screenshots. So the billed image input is not 20 × 1,334; it is 1,334 × (1 + 2 + … + 20) = 280,140 tokens. Add the browser toolset definition, which Anthropic puts at about 6,600 input tokens per request, and 20 requests of that is 132,200 tokens on its own.

Model	Cached run	Uncached run	Equivalent single API call
Claude Sonnet 5.5 ($2/$10 per MTok, cache read 0.05x)	about $0.21	about $1.06	about $0.014
Claude Opus 5.5 ($4/$20 per MTok, cache read 0.05x)	about $0.43	about $2.11	about $0.028
Claude Haiku 5.5 ($0.50/$2.50 per MTok above 100k prompt)	about $0.054	about $0.27	about $0.014

Three things fall out of that table.

First, caching is not an optimisation here, it is the difference between a viable and non-viable cost model. Prompt caching reads on Sonnet 5.5 and Opus 5.5 are billed at 5% of base input price, so a cached browser run costs roughly a fifteenth of the same run uncached.

Second, the cheap model is not the cheap option on real prompts. Haiku 5.5 is priced at $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens, and at $0.50 and $2.50 above that. A 20-turn browser trajectory crosses 100,000 tokens within a few turns, so the long-prompt tier is the tier you pay. On OSWorld, Haiku 4.5 self-reported 50.7% against Sonnet 5’s 81.2%, on different harness revisions.

Third, retries are where budgets die. Failed tasks took nearly twice as many steps as successful ones. At roughly 50% realistic success on live sites, a three-attempt policy costs you the equivalent of about five step-units per eventual success, before you pay for browser infrastructure or proxy bandwidth.

How to build a browser agent that survives a real website

1. Pick the cheapest sufficient model and prove it is sufficient. Run 50 real tasks from your own queue on Haiku 5.5 and Sonnet 5.5 before you commit. The cost difference is roughly 4x; the reliability difference on your specific site may be nothing at all, because form-filling on a legacy portal is a different problem from open-web research.
2. Read the page as a tree before you look at a pixel. Anthropic’s own guidance is that a tree read of a typical page often costs fewer input tokens than a screenshot while giving the model references it can act on immediately. Reserve screenshots for canvas rendering, embedded video, heavily virtualised lists and cross-origin iframes.
3. Cap screenshots and cap images per request. Anthropic’s vision docs warn that a request holding more than 20 images hits a stricter per-side dimension limit, and that a long loop reaches that count within a few dozen turns. Resize to 2000 pixels per side, or prune old screenshots in batches rather than one per turn, because dropping one screenshot every turn changes the prefix and invalidates the cache.
4. Enable prompt caching with one breakpoint after the system prompt and tool definitions, plus up to three more on recent tool results. On Sonnet 5.5 and Opus 5.5 that cuts cached input to one-twentieth of the base input rate.
5. Require confirmation before any irreversible action. Anthropic documents passing credentials to the model inside XML tags like robot_credentials and warns that logged-in browser sessions materially increase prompt-injection risk. OpenAI’s computer use documentation goes further: restrict the environment, treat screen content as untrusted, confirm purchases and destructive changes, and set step, time and cost limits with cancellation.
6. Log in as a dedicated low-privilege account, never a personal one. This is Anthropic’s explicit advice for tasks that cannot avoid an authenticated session, and it is also how you avoid the contract problem described below.
7. Build a deterministic fallback for the stable part of the workflow. Hybrid architectures that plan with an agent and execute deterministic steps are what Anchor Browser claims its “12X faster” design does. Whether or not that claim holds, the pattern is sound: let the model find the button once, then cache the recipe.
8. Test against your own target sites, on a schedule. WebCanvas exists because benchmark pages go stale. Your internal portal will change too, and you want to find out from a monitor rather than from a customer.

What breaks: failure modes and gotchas

CAPTCHAs and bot detection. Every managed browser vendor now sells some form of this. Skyvern bundles a basic CAPTCHA solver on Hobby at $29 per month and an advanced solver on Pro at $149, plus residential proxies and city-level geo-targeting. Steel sells CAPTCHA solves at $3 per 1,000 on its Launch tier and $1 per 1,000 on Scale. Browser Use claims a top score of 90.5% on Stealth Bench V2 and sells cloud browsers from $0.02 per hour. Its own documentation is candid about the limit: no browser configuration guarantees that every CAPTCHA can be avoided or solved. Expect to pay for this as a permanent line item, not a launch-week problem.

Prompt injection through the page itself. Anthropic’s own numbers are the best available evidence. Across 123 test cases covering 29 attack scenarios, browser use without the newer mitigations had a 23.6% attack success rate when deliberately targeted in autonomous mode; with the mitigations it fell to 11.2%. On a challenge set of four browser-specific attack types, including hidden malicious form fields invisible to humans and instructions hidden in the URL text and tab title, mitigations took the rate from 35.7% to 0%. Read that as progress, not as safety. OpenAI’s help centre gives users a concrete instruction that follows directly: avoid vague, open-ended prompts like “check my email and handle everything.”

Session expiry and the named-human problem. These agents log in as a person. Cloudflare’s AI Labyrinth exists because automated visitors reveal themselves by following invisible links into a maze of generated pages, and Cloudflare said at launch that AI crawlers were already generating more than 50 billion requests a day, just under 1% of all requests it sees: a figure from that launch post in March 2025, so roughly 19 months old at the time of writing. The practical consequence for you is that a stuck agent is not merely slow; it is a visible, attributable session from your IP on an account that belongs to a person who can be named.

Pages that change under the agent. Anthropic’s browser use documentation is explicit that element references are best-effort and that highly dynamic pages, including virtualised lists and canvas-rendered interfaces, may not expose stable references, forcing the model back to screenshots and coordinate clicks. It also warns that references go stale silently when a tab navigates or the DOM changes materially, and that the API cannot detect a stale reference.

Drag-and-drop and file upload. Both are weak spots. Anthropic’s drag action takes coordinate pairs and needs careful scaling when screenshots are resized. Its file_upload member is disabled by default because it widens what a manipulated page could make the model do, and enabling it safely requires resolving every path against an allowlisted upload directory, because Claude writes those paths while reading untrusted pages.

Retries that quietly change state. A failed attempt may already have submitted a form. Anthropic’s guidance is that a batch can complete a multistep action within one turn, so if your application asks a human to confirm consequential actions, make that check before each block runs, not after the batch.

The Terms of Service trap that ends projects mid-build

This is the section most “AI browser agent” content skips, and it is the one most likely to kill your project.

Start with the federal law, because it is narrower than vendors imply. In Van Buren v. United States, decided 3 June 2021, the Supreme Court held that someone “exceeds authorized access” when they access a computer with authorisation but then obtain information located in areas off limits to them: files, folders, databases they cannot reach. The Court was explicit that it does not cover improper motives for obtaining information that is otherwise available. The opinion even anticipates the browser-agent case, noting that “many websites, services, and databases… authorize a user’s access only upon his agreement to follow specified terms of service.” A footnote maps the circuit split the case resolved, listing the Second, Fourth, Sixth and Ninth Circuits against the First, Fifth, Seventh and Eleventh.

The Department of Justice’s charging policy, at Justice Manual section 9-48.000 and last updated in May 2022, is blunter. It states that a CFAA prosecution “may not be brought on the theory that a defendant exceeds authorized access solely by violating an access restriction contained in a contractual agreement or term of service with an Internet service provider or web service available to the general public, including public websites.” It adds that the area restriction must be established “in a computational sense, that is, through computer code or configuration, rather than through contracts, terms of service agreements, or employee policies.” One exception matters for your risk assessment: when authors later expressly revoke authorisation through unambiguous written cease-and-desist communications that the defendant receives and understands, the Department will consider the defendant unauthorised from that point onward.

Then the Ninth Circuit applied that logic in hiQ Labs v. LinkedIn, and Judge Edward Chen extended it in Meta Platforms v. Bright Data, No. 3:23-cv-00077-EMC, in the Northern District of California on 23 January 2024. The account of both cases here comes from Quinn Emanuel, which handled the Bright Data summary judgment. According to that client alert, the Bright Data ruling held that Meta’s terms of service may not be construed to prohibit logged-off scraping of publicly available data, that a clause prohibiting scraping in perpetuity even after an account ends is unenforceable, and that a CAPTCHA designed to deter scraping does not convert publicly available information into private information the way a login does. The alert is equally clear about what the court did not decide: it did not address scraping data behind a login screen, and Meta’s tortious interference claim remains pending.

So the CFAA is largely not your problem. Your problem is the next three layers.

Contract. LinkedIn’s User Agreement, effective 3 November 2025, prohibits in its “Don’ts” list developing or using “software, devices, scripts, robots or any other means or processes (such as crawlers, browser plugins and add-ons or any other technology) to scrape or copy the Services,” overriding “any security feature” or bypassing “any access controls or use limits,” and using “bots or other unauthorized automated methods to access the Services.” Read that against a browser agent driving your authenticated Chrome profile and the drafting is uncomfortable. LinkedIn’s Professional Community Policies separately prohibit the “sale of scraped data.” Reddit’s Data API Terms, last revised 20 July 2026, grant a revocable licence, forbid circumventing limits, forbid misrepresenting the user agent or OAuth identity, and require you to delete data not needed for your approved use.

The shared-credentials clause. LinkedIn’s agreement also says you will not share your account and are responsible for anything that happens through it. When an agent logs in as a named human and something goes wrong, the platform’s account is the one that gets actioned.

Technical enforcement, now default. Cloudflare’s 1 July 2026 announcement split AI traffic into three categories and named yours. Search covers crawlers that index your content to answer questions later. Agent covers “automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now,” and Cloudflare’s documentation gives the examples explicitly: chat fetch bots and browser-use agents such as Gemini or Claude driving Chrome. From 15 September 2026, new domains onboarding to Cloudflare receive defaults in which Training and Agent bots are blocked on pages that display ads while Search remains allowed. Customers can opt out. Most will not, because an ad is a signal that the owner meant for a person to land there.

Data protection. If the agent acts inside the EU, you are processing personal data with a named individual’s credentials, on infrastructure you may not control, and you are the controller. That is a question for a data protection lawyer, not for this article. The same applies to whether a given deployment needs a DPIA. I have deliberately not attempted to resolve it.

The bottom line

AI browser agents are genuinely useful, and the current best numbers are respectable: 74.3% on WebArena, 61% on the hardest live benchmark, 85% on OSWorld for frontier models. Use them for supervised, reversible, high-variance tasks where a person is watching.

Do not build a business process that assumes autonomous success. On live sites, most agents still fail seven times in ten, the failures concentrate in the long multi-step tasks that are commercially valuable, and every retry doubles the cost before it improves the outcome. Build the human checkpoint in from the first version rather than retrofitting it after your first incident.

And do the legal work before launch, not after a cease-and-desist arrives. The CFAA is narrow in your favour. Contract law, the CFAA-adjacent torts, the credential-sharing clause in the account you are driving, and Cloudflare’s default block are not.

FAQ

What is the most reliable AI browser agent right now?

No vendor publishes a cross-product leaderboard, so the honest answer is by task type. On the WebArena leaderboard tracked by Steel.dev and updated 11 October 2026, WebTactix on DeepSeek v3.2 leads at 74.3% across 812 tasks. On Online-Mind2Web, which uses 300 tasks on 136 live websites graded by humans, OpenAI Operator leads at 61% and Claude Computer Use 3.7 follows at 56.3%. Treat those two numbers as different measurements of different things, not a single ranking.

How accurate are AI browser agents, really?

Between 30% and 75%, depending entirely on the environment. The best result on the self-hosted WebArena suite is 74.3%. On live websites graded by human annotators, only two of six agents tested by the Online-Mind2Web team exceeded 50%, and the rest sat near 30%. VisualWebArena’s own project page reports 19.78% for the best agent against a human baseline of 88.70%.

Can browser agents log into websites with two-factor authentication?

Partially. Skyvern documents support for QR-based TOTP, email-based and SMS-based 2FA, but lists that capability on its Pro plan at $149 per month and above. Anthropic’s Claude in Chrome documents 1Password sign-in in beta on macOS, where the password manager fills credentials so the model never sees the one-time code. ChatGPT agent instead hands the browser back to you for the login step and then resumes.

Why do browser agents fail so often on real websites?

Because benchmark sites do not change, drift, expire sessions or serve CAPTCHAs, and they rarely log you out mid-task. The Online-Mind2Web team found that failed tasks consumed nearly twice as many action steps as successful ones for Browser Use, Claude Computer Use 3.7 and Operator, driven by repeated actions and unexpected pop-ups. WebCanvas was built specifically to test agents against changing live pages and its best agent reached only 23.1% task success.

Is it legal for an AI browser agent to log into a website using my account?

The US Computer Fraud and Abuse Act is narrower than most operators assume, but contract law and site policy are not. Van Buren v. United States, decided 3 June 2021, held that exceeding authorised access turns on gates-up-or-down, and the Department of Justice’s charging policy states a prosecution may not be brought solely for violating a term of service. That does not help you with the site’s own claims. Get a lawyer to review your specific targets before launch.

How much does one browser agent task cost?

My own calculation from published per-token rates, not a measured benchmark: a 20-turn browser task with one 1280x800 screenshot per turn costs roughly $0.21 on Claude Sonnet 5.5 with aggressive prompt caching, and roughly $1.06 without caching, against about $0.014 for the equivalent single API call. That is roughly 15x to 75x. Add retries, and failed tasks taking about twice the steps of successful ones doubles it again.

Should I use ChatGPT agent, Claude in Chrome, or build my own with Playwright MCP?

Use a hosted product for supervised, one-off tasks you watch. Use Playwright or Playwright MCP plus a frontier model when the workflow is repetitive and you control the target site. ChatGPT agent allows 40 messages per month on Plus and 400 on Pro, with agent mode unavailable on the Free plan. Claude in Chrome ships inside paid Claude plans, with Pro at $20 per month or $17 annually and Max from $100 per month.

What is the difference between a browser use tool and a computer use tool?

A browser use tool drives a browser through the accessibility tree, element references and viewport coordinates. Anthropic’s browser toolset exposes 27 member tools by default and 31 in total, and adds about 6,600 input tokens to every request. The computer use tool drives a whole desktop through screenshots alone, exposes 17 member tools, and adds about 4,500 input tokens per request. For tasks that stay inside webpages, Anthropic says the browser tool is the closer fit.

Can I run browser agents in production without a human in the loop?

Not on the open web, and I would not recommend it. Anthropic’s own red-teaming on Claude in Chrome evaluated 123 test cases across 29 attack scenarios and found a 23.6% attack success rate against browser use without the newer mitigations, reduced to 11.2% with them. OpenAI’s documentation for its computer tool requires confirming consequential actions and bounding the run with step, time or cost limits.

Do I need legal advice before deploying a browser agent?

Before you deploy it against any site that is not yours, yes. This article is research, not legal advice. Cloudflare now classifies browser-use agents as Agent traffic and blocks that category by default on ad-supported pages for new domains from 15 September 2026, which is a contractual and technical signal that target sites are actively resisting this traffic.

---

Sources

• WebArena: A Realistic Web Environment for Building Autonomous Agents (arXiv:2307.13854): 812 tasks, four self-hosted domains, 14.41% GPT-4 baseline versus 78.24% human
• An Illusion of Progress? Assessing the Current State of Web Agents (arXiv:2504.01382): Online-Mind2Web, 300 tasks on 136 live websites, Operator 61%, Claude Computer Use 3.7 56.3%, difficulty breakdowns and step-efficiency findings
• An Illusion of Progress?: full HTML text: the specific prose on step counts, error distribution and WebJudge agreement rates used above
• VisualWebArena project page: 910 tasks, full results table including GPT-4o at 19.78% and the 88.70% human baseline
• VisualWebArena (arXiv:2401.13649): ACL 2024 paper of record
• WebVoyager (arXiv:2401.13919): 643 tasks, 15 live websites, 59.1% original result, 85.3% judge agreement with humans
• WebCanvas: Benchmarking Web Agents in Online Environments (arXiv:2406.12373): Mind2Web-Live, 542 tasks, 2,439 intermediate states, 23.1% task success
• Steel.dev WebArena leaderboard: tracked system results including WebTactix at 74.3%, updated 11 October 2026, with per-row provenance notes
• Steel.dev WebVoyager leaderboard: Browser Use 89.1%, Skyvern 85.85%, Operator 87.0%, Project Mariner 83.5%, with submission dates and harness notes
• Steel.dev OSWorld leaderboard: OSWorld-Verified results, step budgets and self-reported flags
• BenchLM WebArena benchmark page: the candid methodological caveat that fixed self-hosted sites do not test drift, anti-bot systems or login flows
• Awesome Agents web agent benchmark leaderboard: secondary aggregator used only to locate primary sources, not cited for facts
• Browser Use GitHub repository: MIT licence, $0.02 per browser-hour cloud browsers, BU Bench and Stealth Bench descriptions
• Browser Use benchmark repository README: BU Bench V2.1 at 200 tasks, Stealth Bench V1 at 80 tasks, runner configuration and revision history
• Skyvern pricing: Free, Hobby $29/mo, Pro $149/mo and Enterprise credit and concurrency limits
• Skyvern GitHub repository: AGPL-3.0 scope, Playwright-compatible SDK, 2FA methods, password manager integrations
• Skyvern quickstart documentation: run lifecycle states including paused-for-human and timed_out
• Browserbase pricing: free tier limits of 3 concurrent browsers and 1 browser hour, $5 in tokens, 15-minute sessions
• Stagehand | Browserbase: shared Playwright primitives, Shadow DOM and iframe support
• Steel pricing: browser hours $0.10 and $0.08, proxy bandwidth $10 and $6 per GB, CAPTCHA solves $3 and $1 per 1,000, session limits
• Steel documentation: positioning as an open-source browser API with stealth, residential proxies and CAPTCHA solving
• Hyperbrowser pricing: browser hours at $0.10, proxy at $10 per GB, HyperAgent and Browser Use at $0.02 per step, credit tiers
• Airtop pricing: Free, Starter $29/mo, Professional $189/mo, Enterprise $558/mo with session and agent limits
• Anchor pricing: free tier at 5 credits per month and 0.1 credits per task, Starter at $50 per month, and the unverified vendor performance claims
• Anthropic computer use tool documentation: computer_toolset_20260801, 17 member tools, roughly 4,500 token overhead, limitations and security guidance
• Anthropic browser use tool documentation: browser_toolset_20260801, 27 default and 31 total member tools, roughly 6,600 token overhead, stale-reference and iframe caveats
• Anthropic API pricing: per-token rates for Sonnet 5.5, Opus 5.5 and Haiku 5.5, cache multipliers, long-context thresholds, web search pricing, Managed Agents runtime
• Anthropic vision documentation: the width ÷ 28 formula, the 1,000 to 1,800 token screenshot range, and the 20-image per-request limit
• Anthropic plans and pricing: Pro $17 annual or $20 monthly, Max from $100 per month, Claude in Chrome availability by plan
• Claude in Chrome: Get started: permission list, Cowork session behaviour, 1Password sign-in, scheduled and multi-tab features
• Piloting Claude in Chrome: 123 test cases, 29 attack scenarios, 23.6% to 11.2% attack success rate, 35.7% to 0% on browser-specific attacks
• OpenAI pricing: GPT-6 family per-token rates, web search $10 per 1,000 calls, data residency uplift
• ChatGPT agent | OpenAI Help Center: monthly message limits by plan, credit rates, takeover mode, workspace site blocking, read 16 October 2026
• ChatGPT credit rate card: agent mode at 30 credits per message, deep research at 50 credits per task
• Introducing ChatGPT agent: BrowseComp 68.9%, Humanity’s Last Exam 41.6, SpreadsheetBench 45.5% versus Copilot in Excel 20.0%
• ChatGPT agent System Card: unified agentic model description and Preparedness Framework classification
• Computer use | OpenAI API: code execution versus computer tool, and the required safety controls
• ChatGPT Atlas retirement | OpenAI Help Center: confirmation that Atlas is retired and no longer receives updates
• OpenAI is shutting down Atlas (TechCrunch, 9 July 2026): redistribution into the desktop app, Chrome extension and cloud browser
• VAN BUREN v. UNITED STATES (Cornell LII): the gates-up-or-down holding, the terms-of-service passage and the circuit split footnote
• DOJ Justice Manual 9-48.000, Computer Fraud and Abuse Act: charging policy, updated May 2022, including the express bar on ToS-only theories and the cease-and-desist exception
• LinkedIn User Agreement, effective 3 November 2025: section 8.2 prohibitions on scraping software, access-control circumvention and bots, plus the account-sharing clause
• LinkedIn Professional Community Policies: the prohibition on the sale of scraped data
• Reddit Data API Terms, last revised 20 July 2026: revocable licence, user-agent honesty requirement, deletion obligations
• Meta Platforms v. Bright Data: Quinn Emanuel client alert: the 23 January 2024 summary judgment and, crucially, what the court left unanswered
• New options to manage AI traffic, Cloudflare changelog 1 July 2026: the Search, Agent and Training taxonomy and the 15 September 2026 defaults
• Your site, your rules: new AI traffic options for all customers, Cloudflare: the explicit naming of browser-use agents in the Agent category, BotBase, the use robots.txt signal
• Trapping misbehaving bots in an AI Labyrinth, Cloudflare: the 50 billion daily AI crawler requests figure, from the March 2025 launch post
• Amazon Bedrock AgentCore Browser documentation: session isolation, default 15-minute and maximum 8-hour sessions, live view and session replay
• Amazon Bedrock AgentCore pricing: per-second consumption billing, idle time free, 128MB minimum memory billing, Web Search at $7 per 1,000 queries
• microsoft/playwright-mcp: accessibility-tree driving without a vision model, and Microsoft’s note on token cost versus CLI and skills
• ServiceNow BrowserGym: the unified harness covering WebArena, WebArenaVerified, VisualWebArena, WorkArena and AssistantBench, published in TMLR 2025