actionbench methodology
how reliability is measured, what is counted, and what is published.
ActionBench reports reliability per site, action and recipe version, with the raw run data downloadable at /v1/benchmark.
what a run is
A run counts when it finished: succeeded or failed. Denied and canceled runs (policy, dry runs) are not reliability data. A run succeeds only when every postcondition passed.
where runs come from
Two sources, reported separately and together:
- canary: our own scheduled runs with synthetic inputs, against public reads and against sites whose verified owner allowed the
orsel-canaryagent. - client: runs from opted-in clients. Outcomes only; never page content or credentials.
what is reported
Success rate, postcondition pass rate, latency percentiles, results by tier, drift events and the median time from drift to a verified repair, over 24 hours, 7 days and 30 days.
what is not hidden
- Reads and writes are never blended into one number.
- Failures and drift events are published, not filtered.
- A score with few runs says how few. An action with no runs shows no score, not a default.
- Injection-resistance results are reported with and without the trust plane enabled.
limits
These numbers describe our fleet and opted-in clients, not every user's experience. Public agent benchmarks report much lower success on write tasks than on reads; ours will be read the same way.