actionbench methodology

how reliability is measured, what is counted, and what is published.

view benchmark/methodology.md

ActionBench reports reliability per site, action and recipe version, with the raw run data downloadable at /v1/benchmark.

what a run is

A run counts when it finished: succeeded or failed. Denied and canceled runs (policy, dry runs) are not reliability data. A run succeeds only when every postcondition passed.

where runs come from

Two sources, reported separately and together:

  • canary: our own scheduled runs with synthetic inputs, against public reads and against sites whose verified owner allowed the orsel-canary agent.
  • client: runs from opted-in clients. Outcomes only; never page content or credentials.

what is reported

Success rate, postcondition pass rate, latency percentiles, results by tier, drift events and the median time from drift to a verified repair, over 24 hours, 7 days and 30 days.

what is not hidden

  • Reads and writes are never blended into one number.
  • Failures and drift events are published, not filtered.
  • A score with few runs says how few. An action with no runs shows no score, not a default.
  • Injection-resistance results are reported with and without the trust plane enabled.

limits

These numbers describe our fleet and opted-in clients, not every user's experience. Public agent benchmarks report much lower success on write tasks than on reads; ours will be read the same way.