How we measure

Every verdict on an AgentLens audit traces back to a recorded run. This page explains how the numbers are made.

Real agents, exact versions

We run the actual clients — Claude Code and cursor-agent — not simulations or prompted imitations. The exact client version is recorded in every trace, so a result is always attributable to the software that produced it.

Isolated runs

Every run starts in a fresh, isolated sandbox. No shared state between runs, no warm caches, no leftover config: what the agent finds is exactly what a new user of your product would find. Wire logs rotate per run.

Every message recorded

A transparent proxy sits between the agent and your product and records every JSON-RPC message, CLI invocation, and HTTP request — the wire truth, not a summary. When a scenario fails, you can replay exactly what the agent saw and sent at the moment it broke.

Graded mechanically

Verdicts come from a closed set of mechanical assertions declared before the run: a file exists, output contains a value, a tool call succeeded. A scenario passes because its assertions passed — never because the output looked right to a model or a human skimming logs.

Repeated runs, honest statistics

Agents are non-deterministic, so a single run proves nothing. Every scenario runs repeatedly per client, and pass rates ship with 95% Wilson confidence intervals — the interval is the honest width of what we know, and it narrows as runs accumulate.

Failures that aren't yours don't count

When a run dies of client-side or infrastructure causes — the agent crashed, the sandbox lost networking — it leaves the denominator entirely instead of counting against your product. Credential problems are caught by a pre-flight gate before any agent runs, so a bad key never masquerades as a product failure.

Findings are drafted, evidence is primary

Failures become findings drafted by AI and labelled as exactly that. Every claim in a finding links to the trace it cites; the recording is the source of truth, the prose is the summary. Run an audit to see it on your own product →