τ-bench retail.
The design's third falsifier made concrete: a playground-grown program must climb a visible external bench. τ-retail chosen as the fastest real test — multi-turn, tool-using, policy-bound, 115 public tasks. This page shows every asset in the attempt, including what is still unborn. Nothing is hidden, nothing is staged.
the fit that makes it fair: τ scores by comparing the final database state by computation.
their judge is already our court — no model grades anything on either side.
01their world, our organs
| τ-bench concept | playground organ | note |
|---|---|---|
| policy wiki | world.md | the prose every claim must cite |
| mock database (json) | world state | files; jq reads, acts mutate a per-scenario copy |
| agent tools (python) | acts → computed jq transitions | reimplemented in physics; originals kept as specimens, never executed |
| simulated user | caller turns | a rented body on the other side of the desk |
| test task | scenario | world slice + instruction + expected final state — the red is "state diverged" |
| final-state comparison | the diff | computed on both sides; identical epistemology |
02the assets, all of them
Vendored verbatim from github.com/sierra-research/tau-bench @ main, 2026-08-26. Raw bytes served below.
| asset | what | role |
|---|---|---|
| world.md | their retail policy wiki, verbatim | origin of every claim to come |
| world/db/users.json | 500 users | world state |
| world/db/orders.json | 1,000 orders (1.8M) | world state — the thing tasks mutate |
| world/db/products.json | 50 products, variants inside | world state |
| specimens/tasks_test.py | 115 test tasks | scenario source — converted, never run |
| specimens/…16 tools | their tool semantics | reference for the jq reimplementation — parked, like the estate runtime |
| specimens/PROVENANCE.md | source · date · boundary | what was taken, what will never execute here |
All sixteen specimens: calculate · cancel_pending_order · exchange_delivered_order_items · find_user_id_by_email · find_user_id_by_name_zip · get_order_details · get_product_details · get_user_details · list_all_product_types · modify_pending_order_address · modify_pending_order_items · modify_pending_order_payment · modify_user_address · return_delivered_order_items · think · transfer_to_human_agents
03the program — honest state
| organ | state | comes from |
|---|---|---|
names.json | unborn | entities + acts + trace vocabulary, from the wiki and db shape |
claims.json | unborn | conjectured from world.md — every claim cites a wiki fragment |
scenarios/ | unborn | converted from tasks_test; red = "final state diverged" |
| turn-loop contact | owed | contact grows: multi-turn + tool seam; core unchanged (his ruling 10:02) |
| tools as jq | owed | computed world transitions in physics |
| graveyard | empty | fills with the first refused candidate — kept whole, with what killed it |
| first press | owed | 5 tasks first, then the 115; every contact ledgered |
When these rows turn green they will link to bytes, not prose. Until then the mountain is owed, and this page says so. The machinery already running is on /ratchet: two residents clicked, one law seat-written, one false click exposed and fixed.