Methodology
Every number on this site comes from a file results/<id>/<date>.json that validates againstr1-bench.v1. The site is rebuilt from those files; a file that does not validate fails the build.
How a result is produced
- The implementation starts with
AFORO_TEST_MODE=trueandAFORO_CLOCK_FIXED=2027-03-01T12:00:00Zand loads the frozen seedseed-v1.json. - The conformance suite runs black-box against it: Schemathesis (contract), Hurl (acceptance, one file per rule and error case), k6 (race invariants: 300 VUs on a 50-seat event), the security checks and the webhook receiver. Web implementations also run
@aforo/e2e(28 Playwright scenarios, axe, Lighthouse CI). - The load profile is fixed: 200 RPS for 3 minutes (70 % reads, 25 % availability, 5 % bookings). Latency percentiles, errors and memory are recorded.
- Each measurement runs 5 times on GitHub Actions
ubuntu-24.04(4 vCPU); the file holds the median.make benchreproduces it locally. - Factory columns are aggregated from the implementation devlog with the
factory-run.v1schema.
Acceptance criterion
An implementation is conformant with 100 % Hurl, 0 Schemathesis failures, exact race invariants and no ZAP High alert. Its own line coverage must be at least 85 % (80 % for web).
Noise caveat
Shared CI runners are noisy: the same commit can move by 10 to 30 % between days. Runtime numbers are markedindicative. Read them as orders of magnitude and look at the history page of each implementation before drawing a conclusion. Conformance numbers are deterministic and are not affected.
Build order is a bias: later implementations reuse lessons from earlier ones. The factory columns carrybuildOrder so the comparison can account for it.
Factory columns (factory-run.v1)
| Field | Meaning |
|---|---|
tokensInput / tokensOutput | Input and output tokens of the agent sessions that built the implementation (cache reads and creations kept apart). |
costUsd, costBasis | Cost in USD, either `api-equivalent` (list price of the tokens) or `billed`. |
wallClockMinutes, sessions | Elapsed time and number of agent sessions. |
humanInterventions | Times a human had to step in. |
defectsAfterMerge | Defects found after the work was merged. |
toolCalls | Tool calls made by the agents. |
tokenSource, estimated | `ccusage` when tokens were measured, `unavailable` otherwise; estimates are flagged. |
Publishing a result
Add results/<id>/<YYYY-MM-DD>.json through a pull request to r1-aforo-spec. The id must be listed in results/registry.yaml, the folder must match the id and the file name the date.