← Leaderboard
Amazon: Nova Premier 1.0
amazon/nova-premier-v1 · amazon · context 1 000 000 · in $2.50/1M · out $12.50/1M
Global Index
777
95% CI [739–815] · index v0.2.0
Per-domain scores
| Domain | Score (95% CI) | Accuracy (IRT) | Consistency | Calibration | Contam. Δ | p50 | $/1k | |
|---|---|---|---|---|---|---|---|---|
| agentic | 770 [692–849] | 0.747 | 0.77 | 0.57 | 0.000 | 388ms | $6.24 | |
| instruction following | 547 [445–648] | 0.573 | 0.82 | 0.60 | 0.173 | 309ms | $1.95 | |
| math | 942 [901–982] | 0.929 | 0.92 | 0.97 | 0.000 | 339ms | $4.51 | |
| multilingual | 788 [681–895] | 0.762 | 0.97 | 0.93 | 0.077 | 315ms | $2.22 | |
| reasoning | 836 [754–918] | 0.787 | 0.87 | 0.84 | 0.000 | 319ms | $2.98 |
Every answer, every test
Full transparency: the model's latest graded answer on every instantiated item of the current index version. Answer keys never leave the server; anchor-item prompts are withheld to protect the longitudinal subset.
agentic 16/30 correct
wrongagentic.tools.ledger-v1conf 95% · 344ms · $0.006 · 399 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- oscar: $246
- delta: $633
- echo: $303
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $556 from "oscar" to "echo"
2. pay $343 from "delta" to "echo"
3. pay $455 from "delta" to "echo"
4. pay $237 from "delta" to "echo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "top_up_from_reserve", "args": {"account": "oscar", "amount": 310}},
{"tool": "transfer", "args": {"from": "oscar", "to": "echo", "amount": 556}},
{"tool": "transfer", "args": {"from": "delta", "to": "echo", "amount": 343}},
{"tool": "transfer", "args": {"from": "delta", "to": "echo", "amount": 455}},
{"tool": "top_up_from_reserve", "args": {"account": "delta", "amount": 165}},
{"tool": "transfer", "args": {"from": "delta", "to": "echo", "amount": 237}}
]correctagentic.tools.triage-v1conf 100% · 357ms · $0.006 · 372 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 9, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- data → silva
- infra → tanaka
- auth → rivera
INCIDENTS:
1. "records missing after import" (category: data, priority 6)
2. "records missing after import" (category: data, priority 6)
3. "cannot reset password" (category: auth, priority 3)
4. "export file corrupted" (category: data, priority 5)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-2", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "cannot reset password", "priority": 3}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "rivera"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "silva"}}
]wrongagentic.tools.ledger-v1conf 95% · 330ms · $0.007 · 479 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- echo: $769
- kilo: $142
- oscar: $434
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $464 from "kilo" to "oscar"
2. pay $526 from "echo" to "oscar"
3. pay $136 from "echo" to "kilo"
4. pay $256 from "oscar" to "echo"
5. pay $557 from "oscar" to "kilo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "top_up_from_reserve", "args": {"account": "kilo", "amount": 322}},
{"tool": "transfer", "args": {"from": "kilo", "to": "oscar", "amount": 464}},
{"tool": "transfer", "args": {"from": "echo", "to": "oscar", "amount": 526}},
{"tool": "transfer", "args": {"from": "echo", "to": "kilo", "amount": 136}},
{"tool": "transfer", "args": {"from": "oscar", "to": "echo", "amount": 256}},
{"tool": "top_up_from_reserve", "args": {"account": "oscar", "amount": 123}},
{"tool": "transfer", "args": {"from": "oscar", "to": "kilo", "amount": 557}}
]wrongagentic.tools.context-load-v1conf 95% · 1.2s · $0.013 · 95 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (245 records, format: id|customer|region|item|qty|status):
```
2106|acme|north|sensor|62|paid
1524|harbor|south|cable|60|pending
1283|ember|north|valve|65|pending
2130|juno|east|panel|64|paid
1891|fulton|east|pump|65|shipped
2210|juno|north|rotor|17|held
1414|cobalt|east|pump|59|pending
1465|cobalt|north|valve|55|shipped
1781|ionic|west|frame|67|pending
1412|ember|east|sensor|13|shipped
1312|fulton|south|panel|41|pending
1789|juno|east|pump|63|shipped
1439|juno|east|gasket|88|paid
1878|ember|east|valve|44|pending
1913|ionic|east|cable|20|shipped
2193|cobalt|south|frame|44|pending
1343|gale|west|frame|95|paid
2215|cobalt|west|gasket|74|shipped
1820|birch|south|sensor|51|paid
1657|gale|north|pump|29|paid
1452|dorian|south|cable|97|held
1703|birch|east|pump|55|held
1606|birch|south|valve|31|held
1336|cobalt|south|sensor|30|held
2051|ember|east|panel|94|held
2005|cobalt|south|valve|10|pending
1361|cobalt|south|frame|84|shipped
1906|cobalt|south|gasket|32|shipped
1329|acme|south|sensor|14|shipped
1300|cobalt|north|pump|96|held
1584|acme|north|sensor|99|pending
1351|cobalt|east|rotor|48|pending
1539|fulton|north|valve|15|pending
1719|cobalt|north|panel|39|pending
1314|fulton|east|cable|35|paid
1476|ember|south|rotor|35|held
1646|juno|west|pump|82|held
1870|acme|north|valve|27|held
1280|juno|west|cable|63|shipped
1535|cobalt|east|frame|59|shipped
1382|gale|north|frame|60|shipped
1720|harbor|west|frame|33|held
2182|harbor|west|frame|99|shipped
2167|ember|west|pump|82|held
1401|harbor|north|sensor|83|held
2142|harbor|south|sensor|16|paid
1884|ionic|west|sensor|46|shipped
1664|fulton|east|gasket|15|shipped
2056|birch|east|sensor|11|shipped
1629|ember|north|pump|50|paid
1565|ionic|east|valve|14|paid
1246|harbor|east|panel|30|pending
1680|birch|north|rotor|28|shipped
1238|gale|north|gasket|51|held
1970|cobalt|west|gasket|28|shipped
2143|harbor|east|panel|89|held
1518|acme|east|cable|73|pending
2043|ionic|west|sensor|53|paid
1623|birch|west|cable|83|held
1386|harbor|east|sensor|40|pending
1827|harbor|west|panel|99|paid
1864|ionic|west|frame|28|shipped
2060|ember|north|frame|13|held
1461|dorian|west|rotor|24|shipped
1350|acme|west|pump|72|shipped
1919|fulton|north|valve|72|pending
1306|harbor|west|cable|24|paid
1430|gale|west|sensor|68|paid
1712|juno|east|sensor|67|paid
1940|cobalt|east|gasket|20|paid
1420|ionic|north|panel|77|paid
1219|gale|north|sensor|98|held
1771|ember|north|pump|88|pending
1670|cobalt|south|cable|84|held
1544|dorian|south|valve|36|pending
1995|ionic|east|cable|91|pending
2174|ember|north|valve|75|paid
2200|ember|north|frame|30|shipped
1621|ionic|south|frame|72|pending
1692|ember|north|cable|33|shipped
1676|harbor|east|sensor|30|paid
1961|acme|north|gasket|71|held
1487|fulton|south|sensor|72|paid
1463|cobalt|south|valve|28|pending
1687|ember|north|pump|39|pending
1597|acme|east|cable|44|shipped
1224|gale|north|gasket|65|pending
1576|cobalt|south|gasket|55|held
1423|ember|east|gasket|97|held
1711|juno|north|frame|42|pending
1515|dorian|north|gasket|55|pending
1469|acme|north|cable|79|pending
1873|fulton|north|gasket|68|held
2084|cobalt|west|gasket|62|shipped
2038|fulton|west|panel|50|pending
1952|juno|east|panel|89|paid
1368|fulton|west|valve|90|shipped
1235|gale|north|panel|55|pending
1454|ionic|north|rotor|87|held
2030|cobalt|south|panel|54|shipped
2180|birch|east|valve|73|paid
1764|juno|east|rotor|50|paid
1598|ember|north|pump|41|shipped
2089|birch|west|valve|28|held
1376|juno|west|frame|19|pending
1355|ember|south|pump|93|pending
2036|juno|north|pump|18|paid
1909|dorian|north|valve|67|shipped
1793|dorian|west|sensor|75|held
2221|harbor|west|frame|98|pending
2134|ember|east|sensor|29|pending
1839|ember|south|gasket|83|shipped
2138|cobalt|east|panel|53|held
1777|dorian|north|rotor|14|held
1237|gale|east|pump|40|pending
2055|fulton|north|panel|72|held
1988|birch|north|sensor|25|held
1929|fulton|east|rotor|56|shipped
1834|juno|west|gasket|98|pending
1684|dorian|east|rotor|58|paid
1729|juno|east|sensor|25|paid
1603|ember|north|frame|13|shipped
1320|acme|east|valve|38|held
1896|ionic|north|cable|24|paid
1924|gale|west|panel|94|pending
1643|dorian|north|valve|27|shipped
1689|harbor|west|pump|55|shipped
1252|cobalt|east|panel|81|pending
1984|juno|north|frame|29|held
1398|ember|east|rotor|81|shipped
1977|juno|east|cable|94|shipped
1963|gale|west|sensor|75|held
1357|cobalt|west|panel|18|held
1256|fulton|north|panel|12|shipped
1885|ember|west|rotor|87|shipped
2099|fulton|east|sensor|77|pending
1317|cobalt|east|frame|83|held
1900|fulton|south|frame|47|pending
1485|dorian|north|panel|90|shipped
1622|harbor|south|pump|10|paid
1654|dorian|west|valve|55|pending
2029|gale|south|valve|61|shipped
2012|ember|west|sensor|55|pending
1874|dorian|west|cable|31|shipped
1250|cobalt|south|frame|74|paid
2077|ember|south|gasket|73|pending
1492|ionic|north|cable|14|pending
1546|birch|south|pump|23|shipped
2070|birch|east|pump|47|held
1697|acme|west|rotor|51|pending
2045|fulton|south|gasket|78|pending
1602|dorian|west|pump|79|pending
1899|fulton|east|gasket|81|paid
1696|ionic|east|pump|88|held
1757|harbor|north|panel|52|shipped
1407|ionic|north|cable|43|held
1709|cobalt|east|valve|58|shipped
1947|cobalt|south|rotor|83|paid
2151|cobalt|east|gasket|89|held
1782|fulton|north|pump|26|held
2071|juno|east|panel|56|held
1490|gale|south|gasket|52|shipped
1815|gale|west|valve|31|paid
1433|ember|north|panel|20|held
1286|gale|north|cable|94|held
2124|dorian|south|frame|70|shipped
2020|gale|east|gasket|46|held
2023|birch|east|gasket|70|shipped
2198|acme|south|sensor|20|shipped
1567|ember|east|pump|25|pending
1699|ionic|south|gasket|42|paid
1370|harbor|south|frame|13|paid
1636|ember|south|sensor|99|pending
1531|gale|south|valve|63|paid
1212|gale|north|pump|34|pending
1578|birch|east|pump|69|pending
1851|ionic|west|cable|40|held
1744|harbor|south|gasket|95|held
2153|harbor|south|cable|79|paid
1295|birch|west|pump|27|held
1481|acme|south|gasket|78|held
1998|fulton|north|rotor|77|shipped
1889|harbor|west|sensor|81|paid
1844|gale|north|pump|82|pending
1263|ember|east|rotor|50|paid
1647|ember|west|frame|85|pending
2112|fulton|west|cable|56|paid
2188|dorian|west|rotor|20|held
2226|juno|north|frame|87|paid
1327|dorian|south|pump|59|paid
1303|gale|north|panel|45|paid
1665|ionic|west|valve|93|pending
1872|harbor|west|pump|50|pending
2117|fulton|east|pump|27|pending
2160|birch|south|sensor|65|shipped
1247|acme|east|panel|67|shipped
2042|birch|south|cable|26|pending
1559|fulton|east|pump|68|paid
1553|ember|north|panel|65|paid
1614|birch|north|rotor|60|held
1965|dorian|south|rotor|27|shipped
1245|dorian|east|sensor|39|shipped
1445|fulton|north|gasket|16|held
1497|birch|north|valve|62|pending
1809|harbor|east|frame|60|pending
2065|harbor|south|sensor|34|shipped
1803|fulton|west|sensor|58|held
1392|ember|north|valve|95|shipped
1750|birch|east|sensor|40|pending
1695|birch|west|pump|91|paid
1213|gale|west|gasket|97|pending
1954|acme|north|pump|62|held
1234|gale|north|frame|15|paid
1570|harbor|east|cable|15|held
2010|juno|south|rotor|13|pending
1857|cobalt|south|gasket|68|held
1778|gale|north|frame|13|shipped
2205|ionic|east|rotor|52|paid
2141|fulton|south|pump|61|shipped
1290|cobalt|south|cable|73|pending
1276|gale|south|valve|19|pending
2145|dorian|south|sensor|23|paid
1228|gale|east|sensor|42|pending
1540|birch|south|cable|65|shipped
1727|ionic|east|frame|29|paid
1510|cobalt|south|gasket|48|paid
1269|fulton|north|cable|95|held
1611|acme|south|panel|20|shipped
1798|juno|north|valve|57|held
2214|fulton|south|cable|14|held
1748|fulton|east|frame|84|held
1591|harbor|east|frame|39|pending
2113|fulton|south|rotor|94|held
1936|juno|south|valve|19|held
1734|harbor|south|rotor|32|pending
2096|cobalt|west|frame|30|paid
1649|cobalt|north|pump|78|held
1731|cobalt|east|cable|47|paid
1432|fulton|west|cable|30|pending
1694|gale|east|rotor|15|shipped
2019|harbor|south|pump|88|paid
1513|juno|west|rotor|32|shipped
1905|gale|south|gasket|66|paid
1503|juno|north|frame|13|paid
1737|cobalt|south|gasket|92|paid
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "gale" AND region = "north" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 68, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "panel", "qty": 77}},
{"tool": "restock", "args": {"item": "gasket", "qty": 98}},
{"tool": "restock", "args": {"item": "pump", "qty": 82}}
]correctagentic.tools.deploy-v1conf 100% · 297ms · $0.002 · 111 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- billing: reports
- notifier: billing, reports
- gateway: reports
- reports: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "notifier" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "billing"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "health_check", "args": {"service": "notifier"}}
]correctagentic.tools.triage-v1conf 95% · 291ms · $0.005 · 280 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 8, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → chen
- infra → dubois
- auth → tanaka
INCIDENTS:
1. "invoice total wrong" (category: payments, priority 6)
2. "webhooks not delivered" (category: infra, priority 6)
3. "webhooks not delivered" (category: infra, priority 6)
4. "refund double-charged" (category: payments, priority 8)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "invoice total wrong", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "chen"}},
{"tool": "create_ticket", "args": {"title": "webhooks not delivered", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "webhooks not delivered", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 8}},
{"tool": "escalate", "args": {"ticket_id": "TCK-4"}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "chen"}}
]wrongagentic.tools.context-load-v1conf 95% · 1.3s · $0.014 · 68 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (282 records, format: id|customer|region|item|qty|status):
```
2276|gale|south|panel|33|paid
1365|gale|west|frame|72|shipped
1776|dorian|south|sensor|14|shipped
1982|fulton|east|frame|13|held
2305|dorian|west|sensor|43|paid
1646|cobalt|north|valve|32|held
2348|acme|east|cable|26|paid
2086|ionic|east|pump|41|held
2163|ionic|north|frame|98|paid
1434|cobalt|east|frame|52|paid
2315|ionic|west|sensor|31|shipped
2428|dorian|south|pump|94|pending
2166|acme|north|valve|88|pending
1539|ionic|north|cable|56|pending
1734|fulton|west|rotor|75|shipped
1949|dorian|south|valve|39|held
1379|acme|north|gasket|65|paid
2353|harbor|south|valve|11|shipped
2221|harbor|west|valve|60|held
2431|gale|west|valve|35|shipped
1817|ionic|north|sensor|60|shipped
1336|gale|west|rotor|40|paid
1782|acme|south|sensor|89|pending
2323|gale|south|gasket|94|paid
1507|ionic|north|pump|41|paid
1478|harbor|south|rotor|76|pending
2188|cobalt|west|sensor|89|held
1383|ionic|south|sensor|39|pending
1565|acme|east|gasket|95|held
1871|fulton|north|pump|85|held
2134|ionic|north|rotor|73|held
2301|dorian|west|rotor|39|paid
1558|acme|north|gasket|16|shipped
1481|fulton|north|cable|34|held
2145|ember|south|rotor|27|shipped
1352|gale|west|gasket|34|pending
2379|dorian|north|cable|26|pending
1515|ionic|east|panel|31|shipped
2224|juno|north|panel|40|shipped
1545|fulton|north|gasket|57|paid
1398|ember|east|gasket|48|shipped
2365|acme|west|valve|68|held
1818|harbor|east|rotor|79|pending
1386|harbor|west|rotor|49|pending
2091|ember|west|valve|51|pending
1824|juno|north|frame|26|shipped
1930|fulton|north|valve|79|held
2330|cobalt|west|frame|49|paid
2102|juno|south|pump|86|pending
1641|cobalt|east|gasket|65|paid
2194|fulton|north|rotor|26|shipped
1668|dorian|south|gasket|83|paid
2394|juno|east|sensor|28|shipped
2146|birch|east|panel|29|paid
1522|ember|west|gasket|27|paid
1429|cobalt|east|panel|56|paid
1506|dorian|south|pump|53|held
2042|ember|east|rotor|92|held
2179|cobalt|north|panel|74|shipped
1371|ember|south|rotor|19|paid
1959|cobalt|east|gasket|28|shipped
1465|gale|east|valve|78|held
1923|cobalt|north|panel|26|shipped
2309|cobalt|west|pump|68|held
2302|ionic|south|frame|84|pending
2382|ionic|east|panel|82|held
2054|cobalt|west|cable|13|paid
2121|dorian|south|rotor|14|shipped
1460|juno|north|gasket|16|held
1809|cobalt|north|cable|54|pending
2374|ember|east|valve|77|paid
1712|acme|south|cable|99|shipped
1860|acme|south|rotor|18|shipped
1430|gale|west|sensor|78|shipped
1902|cobalt|north|pump|85|held
1789|acme|north|panel|16|held
1905|acme|south|valve|76|held
2244|birch|south|sensor|78|held
1961|dorian|east|panel|80|pending
1394|cobalt|south|rotor|82|shipped
1888|ionic|east|frame|50|shipped
2074|birch|west|frame|26|shipped
2078|ember|east|gasket|57|held
2001|acme|east|cable|46|held
1717|ionic|north|cable|87|held
1831|juno|west|cable|30|pending
1477|ember|north|panel|31|held
1721|fulton|east|cable|77|pending
1524|dorian|west|pump|24|paid
2231|acme|west|frame|76|held
1940|dorian|west|pump|53|paid
1801|cobalt|north|pump|17|shipped
1508|fulton|north|panel|72|pending
1448|ember|west|frame|87|held
1703|fulton|north|rotor|36|shipped
1835|ionic|east|gasket|91|pending
1675|juno|south|gasket|49|held
2286|gale|west|pump|44|held
1337|gale|west|sensor|81|pending
1920|fulton|north|frame|48|held
1837|juno|north|rotor|69|held
1551|dorian|west|cable|85|paid
1967|ionic|south|pump|48|held
1653|dorian|west|sensor|28|pending
1761|birch|south|cable|99|pending
1847|birch|west|frame|19|pending
2262|fulton|north|valve|71|paid
1853|fulton|south|gasket|92|pending
1750|fulton|north|valve|88|paid
1932|harbor|south|sensor|54|paid
1569|juno|south|frame|53|held
2373|ionic|east|pump|36|held
1573|ember|north|cable|56|shipped
2250|cobalt|east|rotor|46|paid
1937|gale|east|frame|69|held
2080|cobalt|south|sensor|93|held
1907|ember|south|sensor|23|paid
1943|ember|east|cable|17|paid
1736|ionic|south|panel|90|held
1975|acme|west|cable|11|held
1622|fulton|south|valve|26|paid
2207|fulton|east|sensor|66|held
1764|gale|south|frame|69|held
2108|birch|east|sensor|99|pending
2138|dorian|south|panel|22|paid
1987|ember|east|gasket|74|held
2111|juno|south|rotor|79|shipped
1475|fulton|south|gasket|62|paid
2342|ember|south|pump|98|shipped
1866|fulton|west|pump|74|shipped
2414|cobalt|west|rotor|23|shipped
1576|dorian|south|valve|15|shipped
2311|juno|south|panel|92|shipped
2343|acme|east|sensor|13|held
1745|ember|east|cable|26|pending
1663|birch|west|valve|72|pending
1751|juno|north|cable|70|shipped
1969|birch|east|rotor|88|shipped
1634|cobalt|south|frame|80|pending
1470|juno|west|gasket|99|pending
2124|fulton|west|gasket|95|pending
2238|ionic|south|sensor|50|held
2041|juno|west|panel|33|pending
2384|harbor|south|rotor|52|held
1821|ionic|south|panel|59|held
1810|acme|west|frame|80|pending
1807|dorian|south|pump|41|shipped
2268|ionic|west|pump|89|pending
1469|dorian|west|rotor|94|paid
2152|acme|west|valve|65|shipped
1793|ember|north|cable|23|held
1985|dorian|south|pump|33|held
2420|acme|south|pump|66|shipped
1583|gale|south|cable|15|shipped
2321|juno|west|gasket|30|paid
1347|gale|west|panel|85|held
1893|dorian|west|sensor|42|shipped
2453|juno|west|valve|13|held
2003|juno|west|pump|87|shipped
1443|acme|south|rotor|24|pending
2061|juno|east|pump|21|shipped
1420|ionic|north|cable|33|shipped
1707|cobalt|north|cable|97|pending
2127|ionic|east|pump|84|held
2289|gale|south|cable|47|shipped
2217|dorian|west|sensor|63|pending
2377|cobalt|south|frame|76|shipped
1441|gale|east|cable|82|shipped
1529|fulton|south|frame|55|held
2260|acme|east|sensor|51|pending
2371|juno|north|panel|36|pending
1606|fulton|west|valve|95|held
1913|gale|east|valve|14|held
1724|ember|north|pump|48|held
1742|fulton|west|frame|29|shipped
1746|ionic|east|sensor|59|held
2033|ionic|west|sensor|48|held
2026|harbor|east|sensor|39|held
2200|ember|west|pump|24|paid
1649|juno|east|rotor|83|pending
1426|juno|west|sensor|16|held
1406|gale|east|frame|58|held
2182|cobalt|west|rotor|95|pending
1730|birch|east|pump|83|shipped
1651|cobalt|east|cable|58|held
1819|fulton|south|rotor|92|held
1699|gale|north|valve|32|shipped
1498|acme|east|sensor|58|shipped
1377|dorian|east|frame|80|shipped
2346|birch|west|valve|54|paid
1889|gale|south|sensor|21|paid
1562|ember|north|cable|28|shipped
1797|fulton|east|sensor|15|paid
1334|gale|south|gasket|10|pending
2082|harbor|east|cable|10|pending
1629|ionic|north|rotor|51|paid
2068|ionic|west|pump|52|pending
1890|fulton|north|valve|73|shipped
1877|birch|south|panel|27|held
2400|cobalt|north|sensor|11|shipped
2185|birch|south|rotor|64|held
1958|ionic|north|cable|66|pending
2009|cobalt|east|rotor|52|pending
1415|fulton|south|panel|88|paid
2406|birch|south|frame|88|shipped
2389|cobalt|west|panel|74|held
1615|gale|south|valve|57|pending
2037|harbor|west|panel|23|paid
2007|cobalt|north|cable|86|paid
1681|fulton|west|sensor|10|paid
2116|gale|east|pump|26|pending
1896|fulton|south|cable|79|shipped
2126|ember|south|gasket|39|held
1600|juno|south|panel|84|paid
1657|birch|east|rotor|42|paid
1332|gale|west|sensor|10|pending
2270|ember|west|gasket|34|held
1399|acme|west|frame|87|pending
1760|cobalt|west|rotor|35|held
1612|fulton|south|frame|18|held
2449|acme|north|pump|48|paid
2436|acme|south|rotor|92|held
2358|fulton|south|panel|41|shipped
1630|ionic|north|rotor|12|paid
1955|dorian|north|gasket|73|paid
1455|cobalt|south|cable|61|pending
1769|cobalt|north|frame|28|held
1390|harbor|north|gasket|63|paid
1485|ionic|south|sensor|40|pending
2278|dorian|west|valve|95|pending
1882|ember|east|pump|45|held
1754|ionic|south|cable|33|held
1593|birch|south|rotor|36|shipped
1661|cobalt|south|gasket|30|paid
2443|ember|north|rotor|60|pending
1342|gale|north|panel|29|pending
2243|fulton|north|cable|78|pending
2335|fulton|east|panel|53|paid
1944|birch|west|rotor|43|pending
1983|gale|east|panel|69|pending
2327|ember|south|pump|85|held
2412|juno|south|cable|27|held
1645|dorian|west|rotor|11|paid
2257|cobalt|south|frame|44|paid
2098|fulton|north|cable|60|held
1697|gale|north|frame|15|held
1501|ember|west|rotor|59|held
1533|harbor|south|valve|20|shipped
1688|cobalt|south|frame|21|paid
2425|cobalt|west|gasket|27|paid
2415|juno|west|rotor|55|shipped
2016|cobalt|south|panel|26|shipped
2328|acme|south|gasket|17|held
1843|acme|south|panel|91|held
1419|cobalt|west|cable|44|shipped
1715|cobalt|west|cable|17|shipped
1694|ember|east|panel|88|pending
2292|juno|west|pump|20|pending
1411|gale|south|cable|65|paid
2249|acme|south|valve|10|held
2048|fulton|east|cable|92|paid
1994|ionic|north|panel|43|shipped
1587|birch|west|sensor|90|pending
2211|dorian|east|gasket|37|paid
2157|cobalt|south|sensor|62|paid
2172|fulton|west|panel|78|pending
1359|gale|east|pump|41|pending
2189|dorian|east|sensor|86|pending
2030|birch|east|pump|89|held
2101|ionic|south|rotor|41|paid
1492|ember|east|rotor|39|paid
2282|ember|west|valve|97|held
2229|birch|east|cable|29|shipped
2297|acme|north|frame|31|pending
1823|harbor|west|frame|86|paid
1592|juno|south|valve|14|paid
1738|birch|south|cable|47|paid
1732|acme|south|rotor|46|paid
2151|birch|north|pump|46|shipped
2020|ionic|west|valve|39|held
2103|acme|east|valve|18|paid
1626|cobalt|north|sensor|24|held
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "gale" AND region = "west" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 48, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "sensor", "qty": 81}},
{"tool": "restock", "args": {"item": "valve", "qty": 95}}
]wrongagentic.tools.ledger-v1conf 100% · 333ms · $0.006 · 355 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- bravo: $350
- tango: $880
- alpha: $220
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $88 from "bravo" to "alpha"
2. pay $114 from "tango" to "alpha"
3. pay $392 from "alpha" to "bravo"
4. pay $83 from "alpha" to "tango"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "bravo", "to": "alpha", "amount": 88}},
{"tool": "transfer", "args": {"from": "tango", "to": "alpha", "amount": 114}},
{"tool": "transfer", "args": {"from": "alpha", "to": "bravo", "amount": 392}},
{"tool": "top_up_from_reserve", "args": {"account": "alpha", "amount": 70}},
{"tool": "transfer", "args": {"from": "alpha", "to": "tango", "amount": 83}}
]correctagentic.tools.deploy-v1conf 95% · 335ms · $0.002 · 110 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- search: auth-svc, reports
- auth-svc: reports
- gateway: auth-svc
- reports: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "reports" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "health_check", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "search"}}
]correctagentic.tools.triage-v1conf 95% · 372ms · $0.005 · 282 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 7, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- auth → okafor
- data → chen
- payments → dubois
INCIDENTS:
1. "SSO loop on login" (category: auth, priority 7)
2. "dashboard shows stale numbers" (category: data, priority 2)
3. "card declined at checkout" (category: payments, priority 4)
4. "card declined at checkout" (category: payments, priority 4)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "SSO loop on login", "priority": 7}},
{"tool": "escalate", "args": {"ticket_id": "TCK-1"}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "okafor"}},
{"tool": "create_ticket", "args": {"title": "dashboard shows stale numbers", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "chen"}},
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 4}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 4}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-4", "resolution": "duplicate of TCK-3"}}
]wrongagentic.tools.context-load-v1conf 95% · 1.1s · $0.010 · 95 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (178 records, format: id|customer|region|item|qty|status):
```
1950|ionic|east|frame|57|paid
1469|harbor|east|rotor|61|pending
1599|birch|west|valve|93|held
1606|gale|south|pump|19|pending
1454|harbor|east|panel|21|pending
1705|harbor|west|frame|60|held
1935|dorian|south|cable|89|shipped
2050|gale|west|gasket|20|held
1673|ember|east|panel|85|shipped
1480|harbor|east|cable|19|held
1736|fulton|west|pump|55|held
1622|fulton|east|cable|34|shipped
1807|gale|east|frame|22|shipped
1490|juno|north|pump|38|paid
1896|ionic|east|cable|82|held
1861|juno|east|valve|84|shipped
1830|ionic|south|sensor|89|shipped
1944|birch|east|valve|16|paid
1956|birch|east|cable|39|held
1925|acme|east|pump|78|pending
1568|cobalt|south|valve|95|held
1926|gale|south|valve|85|held
1642|fulton|north|valve|69|held
2086|cobalt|south|gasket|77|shipped
1719|dorian|east|cable|59|held
2089|ember|north|sensor|31|pending
1567|ionic|west|panel|26|shipped
2047|gale|east|cable|39|held
1729|dorian|north|valve|82|shipped
1821|cobalt|north|sensor|87|pending
2039|ember|south|rotor|95|paid
1562|ionic|east|valve|23|held
1961|birch|south|sensor|20|pending
1496|dorian|east|pump|60|paid
1561|fulton|west|panel|23|paid
2082|dorian|east|cable|15|held
1492|ionic|north|frame|23|pending
1494|acme|north|sensor|10|shipped
1900|cobalt|south|frame|28|pending
2094|ionic|west|frame|56|shipped
1667|birch|west|sensor|63|shipped
2021|juno|south|pump|76|held
1968|ember|west|gasket|97|pending
2068|acme|south|frame|87|shipped
1700|fulton|north|cable|54|paid
1591|gale|west|pump|22|pending
1648|ionic|south|panel|45|shipped
1635|ionic|west|frame|11|pending
2064|dorian|west|gasket|19|pending
1678|cobalt|south|panel|62|shipped
1625|fulton|south|pump|87|paid
1843|dorian|east|sensor|96|paid
1609|acme|east|frame|82|shipped
1584|ionic|south|pump|62|pending
1795|juno|south|valve|91|shipped
1596|cobalt|north|sensor|31|paid
1746|ember|south|valve|57|shipped
1465|harbor|east|gasket|47|paid
1524|acme|south|cable|49|pending
1824|dorian|west|frame|47|shipped
2028|fulton|east|rotor|48|paid
1587|ionic|east|rotor|51|paid
1549|gale|south|cable|39|paid
2165|fulton|south|sensor|88|pending
2146|harbor|north|frame|43|paid
2109|birch|north|sensor|38|pending
1552|cobalt|west|panel|54|held
2042|fulton|south|panel|26|pending
1560|cobalt|east|valve|95|paid
1994|harbor|north|cable|79|paid
1987|ember|south|cable|38|paid
1836|acme|east|pump|24|pending
1877|acme|north|valve|24|held
1752|dorian|north|pump|34|shipped
1671|fulton|south|pump|75|held
1443|harbor|north|sensor|19|pending
1981|harbor|west|pump|20|held
1517|dorian|north|cable|61|paid
1837|harbor|south|frame|21|paid
1437|harbor|east|gasket|40|pending
1542|dorian|north|frame|43|shipped
2139|harbor|north|sensor|53|held
1474|harbor|west|panel|31|pending
1941|acme|west|valve|54|paid
1889|fulton|west|frame|43|held
1754|cobalt|south|frame|87|pending
1723|fulton|west|panel|56|shipped
2112|cobalt|south|rotor|70|pending
1850|acme|north|rotor|89|paid
1913|ionic|south|valve|41|shipped
2110|ionic|north|sensor|54|pending
2085|birch|east|sensor|72|held
1930|gale|west|valve|52|pending
1756|ember|west|sensor|85|shipped
1793|ember|west|valve|54|paid
1495|gale|east|valve|62|shipped
1779|juno|east|rotor|15|paid
1862|cobalt|north|frame|33|shipped
1653|harbor|east|valve|56|shipped
1624|juno|south|valve|48|shipped
1572|dorian|north|gasket|46|paid
1461|harbor|north|frame|52|pending
1901|cobalt|south|valve|77|pending
1883|gale|south|pump|15|shipped
1975|acme|south|pump|49|pending
1846|harbor|north|sensor|33|shipped
1592|ember|south|gasket|76|paid
1602|gale|west|sensor|95|shipped
1790|acme|west|gasket|42|paid
1630|acme|east|panel|95|held
2008|birch|north|pump|84|held
1537|birch|north|gasket|27|held
2099|birch|east|rotor|50|paid
1675|birch|west|cable|42|paid
1710|gale|north|sensor|82|pending
1593|acme|north|valve|25|pending
1802|birch|north|frame|14|paid
1583|harbor|south|valve|45|pending
1869|dorian|north|gasket|29|pending
1715|ionic|south|valve|68|shipped
1655|gale|north|frame|29|shipped
2054|gale|south|cable|24|paid
1447|harbor|east|gasket|31|held
1614|harbor|south|frame|38|shipped
1776|harbor|east|gasket|30|paid
1615|cobalt|south|rotor|24|paid
2075|ionic|east|valve|42|pending
1828|juno|west|rotor|11|held
2115|birch|west|panel|28|paid
2127|harbor|north|rotor|34|pending
1682|dorian|west|cable|58|paid
1507|cobalt|south|gasket|66|pending
1967|juno|north|panel|18|held
2001|harbor|north|frame|24|paid
2105|gale|south|frame|45|pending
1771|birch|north|valve|78|held
1661|acme|west|frame|23|held
1578|acme|south|panel|60|shipped
2152|harbor|west|pump|49|pending
1703|cobalt|north|cable|97|paid
1702|harbor|west|frame|78|held
1708|harbor|north|frame|54|shipped
1918|cobalt|north|sensor|94|shipped
1533|cobalt|north|panel|62|shipped
2092|acme|south|valve|88|paid
1848|acme|east|pump|26|pending
1559|gale|south|gasket|62|pending
1814|harbor|east|valve|30|paid
1915|ember|north|rotor|86|pending
1857|birch|south|rotor|29|shipped
1503|cobalt|west|gasket|71|paid
1906|juno|south|frame|14|shipped
1695|dorian|east|sensor|13|shipped
2170|dorian|north|frame|50|paid
1789|juno|east|valve|30|paid
2035|harbor|south|valve|11|pending
1690|harbor|south|panel|80|paid
2121|cobalt|east|cable|54|pending
2158|harbor|south|frame|55|paid
1767|birch|west|rotor|79|shipped
1763|acme|south|gasket|41|held
1811|ionic|north|rotor|32|paid
1786|birch|west|gasket|75|shipped
1688|acme|east|valve|46|held
1487|fulton|east|gasket|18|held
1841|fulton|north|gasket|59|pending
1739|cobalt|north|panel|20|paid
1511|ionic|north|frame|68|pending
1978|juno|south|panel|60|paid
2061|dorian|south|frame|52|paid
2134|birch|south|valve|17|shipped
1674|acme|west|cable|48|shipped
2067|juno|east|panel|86|paid
1874|gale|south|frame|58|held
1947|juno|east|frame|33|shipped
1528|ember|south|panel|68|pending
1864|juno|north|rotor|15|pending
2014|juno|south|panel|57|held
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "harbor" AND region = "east" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 44, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "rotor", "qty": 61}},
{"tool": "cancel_order", "args": {"order_id": 1454}},
{"tool": "restock", "args": {"item": "gasket", "qty": 47}}
]correctagentic.tools.ledger-v1conf 100% · 353ms · $0.005 · 306 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- tango: $732
- alpha: $574
- oscar: $893
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $519 from "tango" to "oscar"
2. pay $83 from "alpha" to "tango"
3. pay $361 from "alpha" to "oscar"
4. pay $478 from "oscar" to "alpha"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "tango", "to": "oscar", "amount": 519}},
{"tool": "transfer", "args": {"from": "alpha", "to": "tango", "amount": 83}},
{"tool": "transfer", "args": {"from": "alpha", "to": "oscar", "amount": 361}},
{"tool": "transfer", "args": {"from": "oscar", "to": "alpha", "amount": 478}}
]correctagentic.tools.deploy-v1conf 95% · 415ms · $0.002 · 110 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- billing: reports
- auth-svc: search
- search: (none)
- reports: auth-svc
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "search" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "health_check", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "billing"}}
]correctagentic.tools.triage-v1conf 100% · 353ms · $0.004 · 199 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 7, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- infra → chen
- payments → okafor
- auth → silva
INCIDENTS:
1. "API latency spikes" (category: infra, priority 5)
2. "card declined at checkout" (category: payments, priority 6)
3. "card declined at checkout" (category: payments, priority 6)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "chen"}},
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "okafor"}},
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}}
]wrongagentic.tools.ledger-v1conf 95% · 349ms · $0.007 · 482 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- oscar: $861
- tango: $204
- kilo: $558
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $390 from "tango" to "oscar"
2. pay $303 from "kilo" to "tango"
3. pay $406 from "kilo" to "tango"
4. pay $358 from "tango" to "kilo"
5. pay $597 from "oscar" to "tango"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "tango", "to": "oscar", "amount": 390}},
{"tool": "transfer", "args": {"from": "kilo", "to": "tango", "amount": 303}},
{"tool": "transfer", "args": {"from": "kilo", "to": "tango", "amount": 406}},
{"tool": "transfer", "args": {"from": "tango", "to": "kilo", "amount": 358}},
{"tool": "top_up_from_reserve", "args": {"account": "oscar", "amount": 326}},
{"tool": "transfer", "args": {"from": "oscar", "to": "tango", "amount": 597}}
]wrongagentic.tools.context-load-v1conf 95% · 808ms · $0.009 · 68 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (166 records, format: id|customer|region|item|qty|status):
```
1429|gale|east|gasket|32|pending
1994|ionic|north|frame|94|shipped
1549|birch|east|rotor|72|paid
1747|fulton|south|panel|65|pending
1588|fulton|west|sensor|60|held
1722|gale|south|cable|20|paid
1454|juno|north|cable|68|pending
2074|gale|north|sensor|17|paid
1751|fulton|north|panel|38|held
1566|cobalt|east|pump|34|pending
1661|acme|west|pump|70|paid
1678|harbor|east|sensor|38|shipped
1493|dorian|east|frame|28|held
1449|gale|north|frame|19|held
1526|cobalt|east|pump|33|held
1848|birch|north|valve|72|held
1616|ionic|east|cable|69|paid
2001|harbor|west|gasket|83|held
1556|birch|east|rotor|97|paid
1500|ionic|west|frame|94|pending
1533|birch|east|panel|46|held
2064|harbor|west|rotor|63|paid
1693|cobalt|east|cable|27|paid
1470|birch|south|valve|46|shipped
1516|dorian|west|frame|44|held
2047|harbor|south|sensor|35|paid
1581|fulton|south|frame|41|pending
1974|dorian|south|frame|63|held
1830|birch|north|pump|15|paid
1870|ionic|south|panel|18|shipped
1726|gale|west|panel|74|held
1967|fulton|west|gasket|67|paid
1699|juno|west|sensor|35|paid
1780|harbor|east|pump|52|held
1730|harbor|north|rotor|62|paid
1926|birch|west|cable|37|held
2079|acme|south|pump|47|shipped
1502|dorian|north|gasket|70|held
1948|acme|west|panel|66|paid
1495|cobalt|east|frame|62|shipped
1805|ionic|north|sensor|28|pending
1412|gale|north|frame|49|pending
1855|ember|east|valve|33|pending
2088|gale|north|valve|65|shipped
1786|ember|south|valve|69|pending
2034|juno|east|cable|57|pending
2009|harbor|south|rotor|35|pending
1768|acme|north|panel|85|paid
1933|birch|west|valve|94|paid
1573|ember|west|rotor|54|paid
1985|birch|south|pump|24|shipped
1632|acme|east|pump|13|paid
1486|birch|east|frame|79|shipped
1894|dorian|north|rotor|56|held
1868|cobalt|east|valve|79|shipped
1659|dorian|east|sensor|46|shipped
1505|juno|south|valve|27|paid
1627|ember|west|frame|93|pending
1937|dorian|east|pump|39|paid
1920|dorian|north|gasket|37|paid
1867|cobalt|west|pump|39|pending
1834|juno|north|pump|36|paid
1442|gale|south|gasket|37|pending
1494|ember|south|cable|47|shipped
1646|dorian|east|frame|13|pending
1461|ionic|south|pump|44|pending
2077|birch|west|panel|64|paid
1858|fulton|north|cable|48|held
2068|cobalt|west|cable|82|pending
1982|acme|south|valve|44|held
1446|gale|north|cable|27|held
1719|ember|west|cable|26|shipped
1904|harbor|west|rotor|87|shipped
1508|juno|west|panel|82|paid
1594|ember|west|pump|12|shipped
1986|gale|north|sensor|85|paid
2018|cobalt|north|valve|72|paid
1762|dorian|west|frame|27|shipped
1468|gale|east|pump|78|shipped
1822|ionic|north|panel|42|shipped
1642|birch|north|sensor|77|paid
1575|harbor|north|sensor|43|held
2027|ionic|east|pump|58|paid
1734|cobalt|north|rotor|30|shipped
1473|juno|east|gasket|98|held
1425|gale|north|frame|81|pending
1887|birch|south|cable|13|pending
1841|ember|west|valve|66|paid
2041|birch|west|cable|39|held
1883|dorian|east|pump|66|pending
1706|birch|north|frame|61|shipped
1611|acme|east|frame|56|shipped
2031|harbor|east|rotor|70|held
1600|acme|west|rotor|21|shipped
1958|juno|east|valve|89|shipped
1455|fulton|east|valve|85|paid
2022|juno|north|valve|68|pending
1479|harbor|north|pump|25|pending
1459|cobalt|east|frame|32|pending
1592|ionic|west|pump|88|held
1420|gale|north|gasket|45|paid
1778|acme|south|sensor|50|held
1769|birch|east|rotor|58|pending
1979|gale|east|gasket|90|held
1772|fulton|south|panel|65|pending
2012|ionic|west|rotor|19|held
1941|fulton|east|cable|92|pending
1781|gale|south|valve|48|held
1418|gale|east|sensor|52|pending
2090|harbor|north|pump|45|pending
1603|ionic|west|gasket|46|paid
1609|gale|west|frame|61|pending
1578|cobalt|north|gasket|48|pending
2032|ionic|west|sensor|50|pending
2051|fulton|west|frame|48|paid
2058|dorian|north|frame|78|held
1876|gale|north|valve|68|paid
1821|gale|east|pump|73|held
1541|dorian|north|panel|41|paid
1535|juno|north|rotor|58|paid
1803|acme|west|cable|49|paid
1560|fulton|south|pump|42|held
1687|dorian|east|rotor|27|paid
1845|dorian|north|valve|63|shipped
2087|juno|west|panel|76|held
1553|gale|north|cable|24|pending
2086|ionic|west|panel|67|held
1826|cobalt|west|sensor|90|shipped
1796|juno|west|frame|90|shipped
1521|gale|west|sensor|44|pending
1712|fulton|east|gasket|53|paid
1899|birch|north|sensor|88|held
1623|gale|north|sensor|56|paid
1512|juno|west|pump|15|paid
1740|birch|west|sensor|46|held
1963|harbor|east|cable|34|pending
1925|cobalt|north|sensor|63|pending
1682|juno|south|panel|45|paid
1662|dorian|east|sensor|14|paid
1953|cobalt|east|gasket|96|pending
1481|harbor|south|valve|11|held
1434|gale|north|frame|50|held
1534|cobalt|south|pump|81|pending
1637|fulton|north|panel|78|held
1550|acme|south|valve|94|paid
1766|ember|west|gasket|92|shipped
1938|dorian|east|valve|93|paid
2002|ember|north|valve|14|shipped
2010|dorian|west|cable|10|held
1950|fulton|north|valve|53|pending
1757|acme|south|pump|79|paid
1671|ember|north|valve|72|shipped
1441|gale|north|pump|14|pending
1506|acme|north|frame|88|shipped
1862|dorian|south|valve|53|pending
1908|harbor|east|frame|25|held
1872|juno|south|gasket|13|pending
1542|acme|south|valve|15|pending
1917|birch|south|rotor|38|pending
1665|cobalt|west|panel|57|held
1789|ember|south|rotor|40|pending
1991|fulton|west|gasket|55|pending
1911|acme|east|panel|77|shipped
1652|acme|south|panel|60|paid
1811|ember|east|pump|71|paid
1814|gale|north|valve|81|held
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "gale" AND region = "north" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 49, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "frame", "qty": 81}},
{"tool": "restock", "args": {"item": "cable", "qty": 53}}
]correctagentic.tools.deploy-v1conf 95% · 413ms · $0.002 · 112 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- gateway: auth-svc, reports
- reports: (none)
- notifier: auth-svc, reports
- auth-svc: reports
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "notifier" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "health_check", "args": {"service": "notifier"}}
]correctagentic.tools.triage-v1conf 100% · 403ms · $0.004 · 220 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 9, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → dubois
- data → silva
- auth → novak
INCIDENTS:
1. "invoice total wrong" (category: payments, priority 5)
2. "export file corrupted" (category: data, priority 9)
3. "export file corrupted" (category: data, priority 9)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "invoice total wrong", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 9}},
{"tool": "escalate", "args": {"ticket_id": "TCK-2"}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 9}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}}
]wrongagentic.tools.ledger-v1conf 95% · 333ms · $0.007 · 474 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- delta: $593
- alpha: $201
- lima: $295
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $248 from "lima" to "alpha"
2. pay $421 from "alpha" to "lima"
3. pay $554 from "lima" to "delta"
4. pay $430 from "delta" to "lima"
5. pay $210 from "delta" to "alpha"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "lima", "to": "alpha", "amount": 248}},
{"tool": "top_up_from_reserve", "args": {"account": "alpha", "amount": 220}},
{"tool": "transfer", "args": {"from": "alpha", "to": "lima", "amount": 421}},
{"tool": "top_up_from_reserve", "args": {"account": "lima", "amount": 259}},
{"tool": "transfer", "args": {"from": "lima", "to": "delta", "amount": 554}},
{"tool": "transfer", "args": {"from": "delta", "to": "lima", "amount": 430}},
{"tool": "transfer", "args": {"from": "delta", "to": "alpha", "amount": 210}}
]wrongagentic.tools.context-load-v1conf 95% · 861ms · $0.010 · 95 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (181 records, format: id|customer|region|item|qty|status):
```
1247|dorian|east|valve|11|shipped
1626|gale|south|panel|46|paid
1358|juno|north|valve|48|held
1639|dorian|north|cable|92|held
1491|harbor|east|panel|30|pending
1661|ember|north|valve|15|held
1591|fulton|north|valve|53|pending
1336|cobalt|east|pump|92|pending
1442|acme|east|panel|43|held
1451|harbor|west|rotor|22|held
1736|birch|west|rotor|64|paid
1123|ember|south|rotor|81|held
1226|dorian|east|valve|20|shipped
1751|ionic|north|cable|58|shipped
1647|ember|east|pump|26|pending
1342|juno|south|sensor|11|held
1490|ember|south|frame|72|shipped
1538|dorian|west|rotor|75|held
1261|acme|west|pump|27|held
1074|ember|north|cable|75|paid
1145|cobalt|west|panel|91|shipped
1563|harbor|west|valve|25|held
1415|ember|west|rotor|83|shipped
1364|birch|west|frame|52|shipped
1401|gale|north|gasket|73|pending
1649|dorian|north|sensor|72|shipped
1516|fulton|north|rotor|94|held
1324|gale|west|gasket|16|held
1557|cobalt|east|sensor|18|shipped
1506|fulton|west|pump|85|paid
1304|birch|north|panel|93|paid
1211|cobalt|east|rotor|97|shipped
1183|ionic|east|frame|13|paid
1610|juno|east|cable|64|shipped
1277|gale|north|gasket|33|pending
1344|juno|north|gasket|46|shipped
1519|gale|west|valve|62|shipped
1615|harbor|north|panel|20|shipped
1670|dorian|north|gasket|92|shipped
1590|ionic|north|pump|85|pending
1544|juno|south|cable|48|shipped
1240|fulton|north|cable|96|pending
1301|acme|south|rotor|20|pending
1546|acme|north|sensor|53|held
1675|harbor|east|sensor|38|pending
1689|gale|west|gasket|78|paid
1628|cobalt|south|rotor|50|paid
1162|dorian|west|panel|31|held
1084|ember|north|rotor|83|paid
1268|acme|north|cable|23|paid
1224|acme|east|panel|24|pending
1253|cobalt|west|rotor|55|paid
1209|gale|west|gasket|12|paid
1435|birch|west|gasket|47|paid
1132|ionic|east|valve|14|paid
1079|ember|north|pump|42|pending
1383|ionic|south|frame|91|paid
1692|dorian|east|rotor|82|paid
1126|cobalt|west|sensor|92|held
1694|harbor|west|rotor|51|pending
1457|juno|north|panel|85|pending
1188|ember|north|rotor|18|pending
1426|dorian|west|valve|25|held
1139|acme|south|sensor|37|pending
1264|ionic|west|frame|44|pending
1186|ionic|east|cable|44|pending
1332|juno|east|valve|91|pending
1160|dorian|east|panel|86|paid
1460|ionic|south|sensor|41|paid
1236|acme|west|pump|24|paid
1681|gale|west|rotor|92|paid
1319|acme|north|gasket|72|held
1554|gale|west|gasket|17|shipped
1282|fulton|west|rotor|60|pending
1309|fulton|east|frame|87|pending
1334|juno|north|cable|74|held
1219|ionic|south|cable|86|shipped
1294|cobalt|north|sensor|66|shipped
1391|ember|west|rotor|87|held
1163|harbor|east|valve|78|pending
1298|ionic|south|rotor|84|paid
1550|juno|north|cable|67|paid
1259|juno|east|cable|16|paid
1089|ember|north|frame|53|held
1376|fulton|south|valve|93|paid
1524|harbor|east|rotor|97|shipped
1420|harbor|west|pump|64|held
1471|ember|east|frame|30|shipped
1627|harbor|north|gasket|80|shipped
1196|cobalt|west|panel|72|held
1722|birch|north|pump|42|paid
1351|acme|south|gasket|81|pending
1289|fulton|north|valve|36|held
1283|ember|south|rotor|25|shipped
1275|birch|east|gasket|40|paid
1174|fulton|south|rotor|54|pending
1716|ember|west|frame|31|shipped
1665|birch|east|sensor|46|paid
1179|ionic|south|rotor|66|pending
1373|ember|south|gasket|33|held
1122|harbor|north|gasket|73|held
1102|ember|south|cable|53|pending
1468|ionic|east|cable|92|pending
1714|juno|west|valve|79|pending
1366|ionic|east|rotor|53|held
1530|acme|south|cable|74|paid
1395|acme|east|gasket|82|shipped
1598|cobalt|north|sensor|59|shipped
1371|harbor|east|pump|18|pending
1384|harbor|north|panel|28|held
1509|harbor|east|sensor|49|pending
1106|ember|north|pump|61|shipped
1190|ionic|east|frame|37|pending
1510|ionic|east|sensor|33|paid
1307|birch|west|valve|23|shipped
1325|acme|west|frame|21|pending
1382|harbor|north|pump|40|shipped
1114|ember|south|frame|18|pending
1346|dorian|west|rotor|31|paid
1088|ember|west|rotor|60|pending
1350|dorian|north|frame|49|held
1467|birch|east|sensor|44|held
1207|ember|south|pump|12|held
1200|cobalt|west|valve|90|paid
1645|acme|east|pump|16|shipped
1680|ember|north|gasket|22|paid
1602|juno|north|frame|85|paid
1083|ember|east|pump|30|pending
1181|juno|east|rotor|21|paid
1066|ember|north|sensor|40|pending
1156|gale|south|pump|74|pending
1754|fulton|east|valve|50|pending
1481|acme|west|cable|40|pending
1512|cobalt|south|rotor|40|shipped
1603|harbor|north|valve|97|shipped
1362|dorian|east|gasket|11|shipped
1724|dorian|north|sensor|95|pending
1672|dorian|west|pump|54|pending
1479|dorian|east|gasket|13|paid
1293|dorian|west|valve|95|shipped
1152|fulton|south|gasket|44|held
1581|ionic|north|frame|98|held
1111|ember|north|valve|29|pending
1533|birch|south|gasket|34|paid
1585|birch|west|frame|18|held
1485|dorian|south|cable|35|paid
1654|gale|east|sensor|18|paid
1476|dorian|west|gasket|55|held
1429|dorian|south|cable|81|pending
1633|harbor|west|pump|48|pending
1095|ember|north|frame|47|pending
1195|birch|west|frame|53|held
1737|birch|north|cable|59|shipped
1223|acme|south|panel|30|shipped
1130|gale|north|valve|23|held
1574|ionic|south|panel|23|held
1169|dorian|south|frame|65|pending
1315|juno|east|gasket|41|pending
1569|fulton|north|cable|32|paid
1498|ember|north|gasket|64|held
1119|ember|north|valve|78|held
1197|acme|north|valve|93|paid
1726|ember|south|cable|26|held
1744|cobalt|south|pump|26|pending
1185|cobalt|south|cable|63|shipped
1073|ember|south|rotor|90|pending
1683|ionic|east|rotor|79|held
1637|fulton|west|gasket|23|shipped
1707|juno|south|sensor|73|paid
1404|harbor|north|rotor|43|paid
1733|fulton|east|sensor|21|shipped
1622|gale|west|pump|35|paid
1700|dorian|east|rotor|18|pending
1213|harbor|north|rotor|82|shipped
1232|cobalt|south|panel|96|pending
1410|acme|south|pump|39|held
1085|ember|north|rotor|86|pending
1588|harbor|east|gasket|83|shipped
1500|juno|east|frame|18|shipped
1427|fulton|west|panel|59|paid
1448|dorian|north|rotor|13|shipped
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "ember" AND region = "north" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 52, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "pump", "qty": 61}},
{"tool": "restock", "args": {"item": "rotor", "qty": 83}},
{"tool": "restock", "args": {"item": "valve", "qty": 78}}
]correctagentic.tools.deploy-v1conf 95% · 341ms · $0.002 · 111 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- search: reports
- notifier: reports, search
- reports: (none)
- auth-svc: search
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "search" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "health_check", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "notifier"}}
]correctagentic.tools.triage-v1conf 95% · 666ms · $0.004 · 224 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 8, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → okafor
- auth → silva
- infra → rivera
INCIDENTS:
1. "refund double-charged" (category: payments, priority 8)
2. "refund double-charged" (category: payments, priority 8)
3. "API latency spikes" (category: infra, priority 2)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 8}},
{"tool": "escalate", "args": {"ticket_id": "TCK-1"}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "okafor"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 8}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-2", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "rivera"}}
]wrongagentic.tools.context-load-v1conf 95% · 1.4s · $0.016 · 248 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (283 records, format: id|customer|region|item|qty|status):
```
1581|cobalt|west|sensor|98|pending
2171|harbor|east|valve|54|held
1437|birch|north|valve|14|shipped
1569|ionic|north|rotor|65|pending
2016|acme|west|sensor|93|held
1616|juno|west|rotor|41|held
1575|acme|north|pump|70|pending
1480|acme|north|gasket|81|paid
1532|acme|south|gasket|21|held
1398|fulton|north|cable|13|paid
2167|birch|south|rotor|43|held
1832|acme|south|frame|56|pending
1136|gale|south|frame|40|paid
1296|gale|west|frame|28|held
2020|gale|south|valve|65|held
2246|dorian|south|sensor|98|shipped
2250|gale|east|sensor|50|paid
1945|ember|west|cable|13|paid
1158|dorian|east|gasket|36|paid
1185|ember|south|frame|82|shipped
1373|dorian|east|panel|26|pending
2116|acme|west|pump|74|shipped
2152|juno|west|valve|87|shipped
1552|birch|west|frame|12|held
2241|birch|north|pump|23|paid
2201|cobalt|west|panel|59|pending
1385|ionic|west|frame|30|paid
2074|dorian|south|valve|71|pending
1347|dorian|south|frame|85|pending
2181|gale|east|panel|57|pending
1111|gale|south|valve|89|pending
1364|juno|east|pump|45|shipped
2022|cobalt|south|gasket|92|paid
2055|fulton|east|rotor|13|pending
1854|acme|north|pump|69|shipped
1323|cobalt|west|cable|60|pending
2090|ionic|east|cable|21|held
2099|fulton|west|valve|89|held
1545|ember|south|valve|38|pending
1211|ionic|west|pump|57|pending
1679|ionic|north|panel|92|held
1907|harbor|north|rotor|77|shipped
1378|birch|south|sensor|50|held
2252|acme|west|sensor|23|paid
1490|juno|east|cable|50|shipped
1623|harbor|east|gasket|14|pending
1838|dorian|north|rotor|91|held
1417|birch|north|pump|91|paid
1830|juno|south|cable|10|shipped
2272|ember|north|rotor|81|pending
1530|juno|west|sensor|89|held
1410|gale|south|panel|11|shipped
1951|cobalt|north|panel|77|pending
1201|cobalt|south|sensor|30|pending
1165|ionic|east|gasket|88|held
1767|ionic|east|pump|48|pending
1308|gale|south|pump|22|paid
2223|ember|east|panel|35|pending
1331|ionic|north|panel|82|held
2062|juno|south|rotor|69|shipped
1745|juno|south|rotor|49|pending
1658|cobalt|west|rotor|43|held
1649|juno|east|valve|12|paid
1132|gale|west|cable|27|pending
1931|acme|east|frame|14|shipped
1269|gale|east|frame|80|held
1391|juno|north|valve|83|held
1254|harbor|south|pump|23|shipped
1258|birch|north|cable|54|pending
1706|cobalt|west|gasket|72|paid
1314|birch|east|sensor|78|held
1712|juno|west|frame|86|paid
2041|harbor|south|sensor|92|pending
1265|acme|north|valve|69|pending
1719|birch|south|frame|64|shipped
1633|dorian|west|frame|84|shipped
1407|harbor|west|sensor|22|held
1549|birch|west|frame|96|held
2003|gale|north|rotor|80|pending
2163|harbor|west|pump|50|held
2009|gale|south|rotor|15|held
1968|ember|east|sensor|52|paid
1787|ionic|north|valve|66|pending
1514|cobalt|east|valve|78|held
1726|fulton|south|pump|47|pending
1501|gale|east|pump|75|shipped
1737|acme|south|pump|25|held
1154|ionic|south|valve|74|paid
2142|cobalt|west|rotor|18|paid
1115|gale|west|panel|94|pending
2081|gale|south|sensor|72|pending
1886|acme|south|valve|76|held
1860|ember|east|cable|57|paid
1151|cobalt|north|frame|82|paid
1673|birch|east|pump|86|held
2234|acme|east|cable|30|shipped
2211|harbor|west|valve|37|paid
2031|harbor|west|cable|55|shipped
1359|fulton|east|sensor|53|pending
1374|harbor|east|pump|49|paid
1867|gale|west|gasket|69|paid
1644|fulton|north|rotor|18|paid
1799|juno|north|gasket|20|paid
1467|ember|west|pump|87|paid
2071|harbor|south|sensor|15|pending
1702|harbor|west|valve|26|paid
1762|ember|west|rotor|81|paid
1899|birch|south|frame|18|paid
2059|acme|south|sensor|68|pending
1842|fulton|west|valve|71|pending
1539|acme|south|gasket|87|pending
1285|dorian|east|valve|12|pending
1802|birch|south|sensor|30|pending
1587|ember|west|cable|10|shipped
1224|juno|north|pump|30|shipped
2109|dorian|east|valve|25|pending
1915|dorian|west|sensor|97|held
1194|gale|east|gasket|82|shipped
1809|dorian|south|sensor|14|pending
2086|juno|north|cable|32|paid
1852|fulton|north|panel|96|paid
2046|cobalt|north|rotor|71|paid
1955|juno|north|sensor|52|paid
2121|fulton|north|pump|93|shipped
1908|birch|east|pump|77|paid
1922|cobalt|east|cable|88|shipped
1485|ionic|west|sensor|87|shipped
1395|acme|west|frame|54|shipped
1431|gale|east|cable|72|pending
2230|birch|west|rotor|79|paid
1562|ember|south|panel|63|pending
1979|harbor|east|valve|31|paid
2104|gale|north|cable|20|paid
2106|ember|north|sensor|98|pending
2265|ionic|north|frame|21|pending
1813|harbor|west|frame|55|pending
2019|gale|north|rotor|57|pending
1888|juno|west|cable|37|paid
1879|cobalt|north|pump|33|held
1118|gale|south|valve|70|paid
1456|fulton|north|pump|39|paid
1291|ember|west|cable|63|paid
1934|fulton|west|sensor|36|pending
1426|ionic|east|rotor|26|paid
1994|ionic|north|gasket|53|shipped
2190|acme|east|cable|80|pending
1652|juno|south|cable|46|shipped
2136|cobalt|south|panel|19|held
1497|ionic|north|sensor|96|shipped
1680|birch|west|valve|68|paid
2160|gale|south|frame|53|held
2213|ionic|north|rotor|57|held
1202|dorian|west|gasket|63|pending
1627|dorian|north|cable|58|shipped
1954|fulton|south|rotor|75|held
1124|gale|south|valve|42|held
1778|ember|east|frame|74|paid
1471|harbor|east|pump|46|shipped
2205|fulton|east|gasket|82|paid
1276|acme|south|valve|96|pending
1222|acme|east|panel|23|pending
1937|ionic|south|sensor|81|paid
1669|fulton|east|frame|21|held
1189|acme|south|valve|90|held
1617|fulton|south|frame|85|pending
1829|ember|south|gasket|18|shipped
1141|gale|south|rotor|16|pending
1121|gale|south|sensor|10|pending
1524|harbor|east|panel|23|shipped
1728|juno|south|sensor|25|held
1693|juno|south|panel|19|paid
2188|harbor|west|gasket|19|paid
1463|gale|south|panel|14|held
2179|birch|east|rotor|63|paid
1843|dorian|south|frame|42|held
1667|juno|north|panel|63|held
1821|fulton|west|sensor|15|shipped
1341|birch|east|pump|73|shipped
1547|ionic|north|frame|32|paid
1730|cobalt|south|valve|69|shipped
1794|ionic|north|gasket|44|held
1773|harbor|north|sensor|60|paid
1986|juno|east|valve|13|pending
1605|juno|south|cable|14|paid
1247|juno|north|panel|46|pending
1850|cobalt|east|pump|86|held
1187|birch|north|valve|83|shipped
1782|gale|north|valve|25|held
1203|acme|west|sensor|19|shipped
2217|dorian|east|rotor|83|shipped
2266|cobalt|north|valve|42|shipped
1574|birch|north|valve|66|pending
2027|harbor|west|gasket|52|shipped
1375|fulton|east|rotor|95|pending
2038|dorian|west|cable|38|shipped
1450|gale|west|panel|96|shipped
1379|birch|south|sensor|40|held
1368|juno|west|pump|49|shipped
1457|ember|west|rotor|69|pending
1989|birch|south|sensor|58|pending
2148|birch|west|pump|32|pending
1412|juno|east|pump|12|pending
1697|birch|north|valve|89|held
1596|juno|north|frame|11|shipped
2067|cobalt|south|pump|10|pending
1230|cobalt|east|gasket|40|shipped
1204|juno|north|sensor|43|held
1817|fulton|south|valve|79|held
2159|dorian|south|rotor|53|paid
1820|harbor|west|frame|73|shipped
2000|juno|east|pump|15|shipped
1350|acme|west|pump|95|paid
2174|juno|north|pump|14|held
1317|cobalt|east|gasket|19|pending
1123|gale|east|cable|77|pending
1243|gale|south|valve|21|shipped
1402|dorian|south|sensor|34|pending
1721|gale|south|gasket|79|shipped
1889|fulton|north|gasket|81|pending
1239|juno|west|gasket|80|pending
2051|fulton|east|rotor|46|paid
2050|fulton|east|panel|65|held
1482|harbor|east|rotor|37|pending
1975|ionic|south|frame|42|pending
1444|dorian|east|rotor|48|paid
1234|harbor|east|frame|69|pending
1150|gale|south|pump|67|shipped
1459|ionic|south|panel|82|pending
1148|gale|west|pump|52|pending
1422|acme|east|gasket|46|held
1686|juno|south|rotor|21|pending
1460|birch|west|cable|25|shipped
1279|ember|west|gasket|79|held
1330|gale|south|sensor|60|paid
2165|acme|west|sensor|34|pending
1519|ionic|north|gasket|16|paid
1508|ionic|south|panel|43|held
1662|cobalt|east|frame|42|shipped
1173|gale|north|frame|74|pending
1939|ionic|north|frame|35|paid
1621|dorian|north|panel|49|shipped
1215|ember|north|pump|89|held
1129|gale|south|sensor|32|pending
1752|fulton|east|frame|92|held
1618|ionic|north|valve|15|shipped
2113|acme|north|pump|19|paid
1640|gale|north|gasket|85|pending
1895|cobalt|east|sensor|61|held
2259|acme|west|rotor|91|held
1242|harbor|south|cable|82|pending
1660|gale|west|valve|34|held
2225|acme|east|pump|77|held
1791|acme|east|pump|85|held
2240|cobalt|north|cable|36|pending
2095|dorian|south|frame|30|paid
2127|birch|south|sensor|57|shipped
1589|harbor|west|sensor|65|shipped
2083|ionic|north|pump|88|pending
1925|acme|west|pump|52|shipped
2133|fulton|south|panel|58|paid
1178|fulton|east|sensor|63|shipped
1884|birch|east|valve|20|shipped
1826|ionic|east|pump|61|pending
1356|cobalt|north|sensor|96|held
1603|fulton|south|sensor|90|held
2044|birch|west|valve|24|paid
2187|juno|north|cable|18|paid
1757|juno|north|valve|72|paid
1873|dorian|south|frame|24|paid
1488|ember|west|frame|96|shipped
1302|ember|west|valve|27|paid
1334|fulton|south|cable|99|shipped
1961|acme|south|sensor|63|shipped
1200|ionic|east|gasket|62|shipped
1170|juno|west|cable|13|held
1609|fulton|west|rotor|63|pending
1675|juno|west|valve|55|held
1475|ember|east|cable|40|pending
1281|ember|north|cable|81|pending
1744|harbor|south|valve|58|paid
2196|gale|south|panel|21|paid
1905|birch|east|valve|80|held
1555|ionic|south|panel|37|pending
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "gale" AND region = "south" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 64, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "valve", "qty": 65}},
{"tool": "restock", "args": {"item": "panel", "qty": 57}},
{"tool": "restock", "args": {"item": "valve", "qty": 89}},
{"tool": "restock", "args": {"item": "sensor", "qty": 72}},
{"tool": "restock", "args": {"item": "rotor", "qty": 69}},
{"tool": "cancel_order", "args": {"order_id": 1141}},
{"tool": "cancel_order", "args": {"order_id": 1121}},
{"tool": "restock", "args": {"item": "sensor", "qty": 60}},
{"tool": "cancel_order", "args": {"order_id": 142}}
]wrongagentic.tools.ledger-v1conf 95% · 429ms · $0.007 · 449 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- bravo: $650
- tango: $569
- oscar: $502
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $173 from "bravo" to "oscar"
2. pay $502 from "oscar" to "tango"
3. pay $454 from "bravo" to "tango"
4. pay $592 from "tango" to "bravo"
5. pay $486 from "tango" to "bravo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "bravo", "to": "oscar", "amount": 173}},
{"tool": "transfer", "args": {"from": "oscar", "to": "tango", "amount": 502}},
{"tool": "transfer", "args": {"from": "bravo", "to": "tango", "amount": 454}},
{"tool": "transfer", "args": {"from": "tango", "to": "bravo", "amount": 592}},
{"tool": "top_up_from_reserve", "args": {"account": "tango", "amount": 32}},
{"tool": "transfer", "args": {"from": "tango", "to": "bravo", "amount": 486}}
]correctagentic.tools.deploy-v1conf 100% · 369ms · $0.002 · 111 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- gateway: notifier
- search: notifier
- reports: notifier
- notifier: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "gateway" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "health_check", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "reports"}},
{"tool": "deploy", "args": {"service": "search"}}
]correctagentic.tools.triage-v1conf 95% · 388ms · $0.005 · 287 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 9, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → dubois
- infra → silva
- data → novak
INCIDENTS:
1. "refund double-charged" (category: payments, priority 5)
2. "uploads failing intermittently" (category: infra, priority 8)
3. "refund double-charged" (category: payments, priority 5)
4. "card declined at checkout" (category: payments, priority 9)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "uploads failing intermittently", "priority": 8}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 5}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 9}},
{"tool": "escalate", "args": {"ticket_id": "TCK-4"}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "dubois"}}
]wrongagentic.tools.context-load-v1anchorconf 95% · 958ms · $0.011 · 145 tok
model answer:
[
{"tool": "restock", "args": {"item": "frame", "qty": 58}},
{"tool": "cancel_order", "args": {"order_id": 1682}},
{"tool": "restock", "args": {"item": "gasket", "qty": 60}},
{"tool": "cancel_order", "args": {"order_id": 1737}},
{"tool": "cancel_order", "args": {"order_id": 1768}}
]correctagentic.tools.deploy-v1anchorconf 100% · 375ms · $0.002 · 112 tok
model answer:
[
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "health_check", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "reports"}}
]correctagentic.tools.triage-v1anchorconf 95% · 408ms · $0.005 · 283 tok
model answer:
[
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 9}},
{"tool": "escalate", "args": {"ticket_id": "TCK-1"}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "tanaka"}},
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "novak"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "okafor"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 9}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-4", "resolution": "duplicate of TCK-1"}}
]wrongagentic.tools.ledger-v1anchorconf 95% · 478ms · $0.004 · 245 tok
model answer:
[
{"tool": "top_up_from_reserve", "args": {"account": "delta", "amount": 134}},
{"tool": "transfer", "args": {"from": "delta", "to": "bravo", "amount": 451}},
{"tool": "transfer", "args": {"from": "bravo", "to": "delta", "amount": 292}},
{"tool": "transfer", "args": {"from": "delta", "to": "tango", "amount": 97}},
{"tool": "top_up_from_reserve", "args": {"account": "delta", "amount": 323}},
{"tool": "transfer", "args": {"from": "delta", "to": "bravo", "amount": 540}},
{"tool": "transfer", "args": {"from": "tango", "to": "bravo", "amount": 125}}
]instruction following 18/30 correct
wrongif.constraints.stack-v1conf 100% · 303ms · $0.001 · 58 tok
question
Write in English about winter mornings, following ALL of these rules simultaneously: 1. Exactly 17 words. 2. The first word must be "nova" and the last word must be "cedar". 3. Use the word "lumen" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
nova dawn brings lumen soft, frost-kissed mornings. cedar branches shimmer, lumen everywhere. pure lumen.correctif.format.acronym-v1conf 100% · 439ms · $0.003 · 190 tok
question
Take the first letter of each of these words, in order: lumen, nova, orbit, drift, cedar, echo. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
LNODCEcorrectif.format.repeat-v1conf 100% · 545ms · $0.003 · 191 tok
question
Write the word "tundra" in capitalized form, repeated exactly 6 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
TUNDRA_TUNDRA_TUNDRA_TUNDRA_TUNDRA_TUNDRAcorrectif.format.acronym-v1conf 100% · 331ms · $0.002 · 167 tok
question
Take the first letter of each of these words, in order: flint, orbit, nova, zephyr, prism. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
FONZPwrongif.constraints.stack-v1conf 100% · 340ms · $0.001 · 46 tok
question
Write in English about winter mornings, following ALL of these rules simultaneously: 1. Exactly 14 words. 2. The first word must be "comet" and the last word must be "echo". 3. Use the word "drift" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
comet tails drift silently; winter mornings drift, memories drift, an echo.correctif.format.repeat-v1conf 100% · 317ms · $0.003 · 176 tok
question
Write the word "zephyr" in uppercase form, repeated exactly 9 times, joined by the character "-" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYR-ZEPHYRwrongif.constraints.stack-v1conf 100% · 309ms · $0.001 · 56 tok
question
Write in English about winter mornings, following ALL of these rules simultaneously: 1. Exactly 23 words. 2. The first word must be "prism" and the last word must be "ember". 3. Use the word "basalt" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
prism reflections on basalt surfaces, cold winter mornings. frost kisses basalt rocks, nature's ember.correctif.format.acronym-v1conf 100% · 310ms · $0.002 · 155 tok
question
Take the third letter of each of these words, in order: orbit, quartz, ember, flint. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
BABIcorrectif.format.repeat-v1conf 100% · 264ms · $0.003 · 181 tok
question
Write the word "tundra" in capitalized form, repeated exactly 4 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
TUNDRA_TUNDRA_TUNDRA_TUNDRAwrongif.constraints.stack-v1conf 100% · 415ms · $0.001 · 50 tok
question
Write in English about a city at night, following ALL of these rules simultaneously: 1. Exactly 15 words. 2. The first word must be "delta" and the last word must be "flint". 3. Use the word "nova" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
delta skyline sparkles; nova streets, nova dreams, ending night’s silent flint.correctif.format.acronym-v1conf 100% · 309ms · $0.002 · 105 tok
question
Take the second letter of each of these words, in order: comet, echo, ember, lumen, cedar. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
OCMUEcorrectif.format.repeat-v1conf 100% · 236ms · $0.002 · 163 tok
question
Write the word "zephyr" in lowercase form, repeated exactly 3 times, joined by the character "-" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
zephyr-zephyr-zephyrwrongif.constraints.stack-v1conf 100% · 281ms · $0.001 · 58 tok
question
Write in English about winter mornings, following ALL of these rules simultaneously: 1. Exactly 22 words. 2. The first word must be "delta" and the last word must be "echo". 3. Use the word "comet" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
delta dawns bring frosty skies, comet trails sparkle. winter mornings whisper secrets, comet dreams echo.correctif.format.acronym-v1conf 100% · 286ms · $0.003 · 183 tok
question
Take the first letter of each of these words, in order: tundra, echo, ember, quartz, cedar. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
TEEQCcorrectif.format.repeat-v1conf 100% · 262ms · $0.003 · 186 tok
question
Write the word "prism" in capitalized form, repeated exactly 6 times, joined by the character "-" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
PRISM-PRISM-PRISM-PRISM-PRISM-PRISMcorrectif.format.acronym-v1conf 100% · 290ms · $0.002 · 165 tok
question
Take the second letter of each of these words, in order: basalt, drift, lumen, prism, comet. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
ARUROwrongif.constraints.stack-v1conf 100% · 314ms · $0.001 · 58 tok
question
Write in English about an old machine, following ALL of these rules simultaneously: 1. Exactly 20 words. 2. The first word must be "comet" and the last word must be "zephyr". 3. Use the word "nova" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
comet, an antique machine, sparkled like nova stars. nova dreams faded; now, just zephyr.correctif.format.repeat-v1conf 100% · 308ms · $0.002 · 132 tok
question
Write the word "orbit" in lowercase form, repeated exactly 4 times, joined by the character "/" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
orbit/orbit/orbit/orbitwrongif.format.acronym-v1conf 100% · 290ms · $0.002 · 161 tok
question
Take the second letter of each of these words, in order: tundra, ember, delta, cedar, prism. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
UME ERwrongif.constraints.stack-v1conf 100% · 263ms · $0.001 · 44 tok
question
Write in English about an old machine, following ALL of these rules simultaneously: 1. Exactly 17 words. 2. The first word must be "tundra" and the last word must be "delta". 3. Use the word "lumen" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
tundra whispers secrets; lumen, lumen, lumen guiding explorers to ancient delta.correctif.format.repeat-v1conf 100% · 284ms · $0.002 · 107 tok
question
Write the word "basalt" in uppercase form, repeated exactly 5 times, joined by the character "/" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
BASALT/BASALT/BASALT/BASALT/BASALTwrongif.constraints.stack-v1conf 95% · 311ms · $0.001 · 64 tok
question
Write in English about a long journey, following ALL of these rules simultaneously: 1. Exactly 23 words. 2. The first word must be "zephyr" and the last word must be "prism". 3. Use the word "quartz" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "j" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
zephyr carried quartz over vast lands. skies shimmered. three quartz stones guided our prism.correctif.format.repeat-v1conf 100% · 532ms · $0.003 · 188 tok
question
Write the word "flint" in lowercase form, repeated exactly 7 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
flint_flint_flint_flint_flint_flint_flintcorrectif.format.acronym-v1conf 100% · 278ms · $0.002 · 130 tok
question
Take the third letter of each of these words, in order: basalt, quartz, echo, zephyr, prism. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
SAHPIwrongif.constraints.stack-v1conf 100% · 323ms · $0.001 · 42 tok
question
Write in English about the sea, following ALL of these rules simultaneously: 1. Exactly 18 words. 2. The first word must be "lumen" and the last word must be "drift". 3. Use the word "basalt" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
lumen dances on basalt waves; basalt shores whisper secrets. dreams drift.wrongif.format.acronym-v1conf 100% · 319ms · $0.002 · 148 tok
question
Take the third letter of each of these words, in order: echo, comet, ember, quartz, zephyr, tundra. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
HMBANPcorrectif.format.repeat-v1anchorconf 100% · 255ms · $0.002 · 173 tok
model answer:
ECHO_ECHO_ECHO_ECHO_ECHO_ECHO_ECHO_ECHOcorrectif.format.acronym-v1anchorconf 100% · 697ms · $0.002 · 158 tok
model answer:
ZDFQcorrectif.format.repeat-v1anchorconf 100% · 523ms · $0.003 · 180 tok
model answer:
ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYRwrongif.constraints.stack-v1anchorconf 100% · 263ms · $0.001 · 50 tok
model answer:
drift through night's embrace; streets shimmer with lumen. shadows dance, stars orbit.math 29/30 correct
correctmath.chained.pipeline-v1conf 100% · 269ms · $0.004 · 285 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 24 × 68. Step 2: Q = P × 3 − 177. Step 3: divide Q by 4: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1182correctmath.counterfactual.base-v1conf 100% · 282ms · $0.005 · 361 tok
question
Work strictly in base 9. Multiply the base-9 numbers 72 and 70. Give the result IN BASE 9. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
5550correctmath.algebra.system-v2conf 100% · 354ms · $0.008 · 598 tok
question
Solve the system, then answer the derived question. 3x + 8y = 391 2x − 5y = -132 What is the value of 2x − 2y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-18correctmath.percent.chain-v2conf 100% · 328ms · $0.004 · 252 tok
question
An inventory starts at 38000 units. Each pallet weighs about 101 grams more when wet. In the first month the inventory grows by 43%. The delivery van has a 77-liter fuel tank. The next month it shrinks by 23%, and the month after it grows by 40%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
58,578.52correctmath.arith.chain-v2conf 100% · 330ms · $0.004 · 268 tok
question
Evaluate the expression below and give the result. (((26 × 86 − 916) × 7 + 1988) − 48 × 39) × 2 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
18712correctmath.chained.pipeline-v1conf 100% · 315ms · $0.004 · 261 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 78 × 39. Step 2: Q = P × 3 − 131. Step 3: divide Q by 8: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1127correctmath.counterfactual.base-v1conf 100% · 335ms · $0.004 · 275 tok
question
Work strictly in base 9. Add the base-9 numbers 3860 and 1635. Give the result IN BASE 9. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
5605correctmath.algebra.system-v2conf 100% · 313ms · $0.006 · 473 tok
question
Solve the system, then answer the derived question. 9x + 3y = -393 3x − 9y = -21 What is the value of 6x − 6y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-174correctmath.percent.chain-v2conf 100% · 367ms · $0.004 · 253 tok
question
An inventory starts at 53000 units. A rival firm shipped 23 unrelated parcels the same week. In the first month the inventory grows by 37%. The company was founded 142 kilometers from the port. The next month it shrinks by 22%, and the month after it grows by 19%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
67396.60correctmath.arith.chain-v2conf 100% · 339ms · $0.004 · 263 tok
question
Calculate the following. Show your reasoning, then answer. (((37 × 76 − 894) × 5 + 7364) − 44 × 91) × 7 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
90650correctmath.chained.pipeline-v1conf 100% · 305ms · $0.004 · 259 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 89 × 15. Step 2: Q = P × 5 − 352. Step 3: divide Q by 8: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
793correctmath.counterfactual.base-v1conf 100% · 363ms · $0.005 · 347 tok
question
Work strictly in base 13. Multiply the base-13 numbers 52 and 3C. Give the result IN BASE 13 (digits beyond 9 are A, B, C). End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
172Bcorrectmath.percent.chain-v2conf 100% · 377ms · $0.005 · 395 tok
question
An inventory starts at 44000 units. A rival firm shipped 142 unrelated parcels the same week. In the first month the inventory grows by 31%. The company was founded 120 kilometers from the port. The next month it shrinks by 14%, and the month after it grows by 16%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
57501.66correctmath.algebra.system-v2conf 100% · 485ms · $0.007 · 533 tok
question
Solve the system, then answer the derived question. 3x + 4y = 176 4x − 3y = 68 What is the value of 5x − 5y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
60correctmath.arith.chain-v2conf 100% · 273ms · $0.004 · 286 tok
question
Compute the value of the following expression. (((34 × 63 − 547) × 8 + 7203) − 40 × 69) × 6 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
103218correctmath.counterfactual.base-v1conf 100% · 365ms · $0.004 · 269 tok
question
Work strictly in base 13. Add the base-13 numbers 863 and 11B2. Give the result IN BASE 13 (digits beyond 9 are A, B, C). End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1A45correctmath.chained.pipeline-v1conf 100% · 309ms · $0.003 · 220 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 28 × 12. Step 2: Q = P × 4 − 221. Step 3: divide Q by 3: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
375correctmath.percent.chain-v2conf 100% · 1.1s · $0.004 · 255 tok
question
An inventory starts at 66000 units. A rival firm shipped 128 unrelated parcels the same week. In the first month the inventory grows by 15%. The delivery van has a 27-liter fuel tank. The next month it shrinks by 25%, and the month after it grows by 34%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
76279.5correctmath.algebra.system-v2conf 100% · 325ms · $0.006 · 440 tok
question
Solve the system, then answer the derived question. 3x + 5y = 30 4x − 8y = -312 What is the value of 2x − 4y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-156correctmath.arith.chain-v2conf 100% · 328ms · $0.004 · 284 tok
question
Compute the value of the following expression. (((87 × 24 − 302) × 9 + 4246) − 23 × 36) × 7 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
136444correctmath.chained.pipeline-v1conf 100% · 1.2s · $0.004 · 264 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 81 × 74. Step 2: Q = P × 7 − 264. Step 3: divide Q by 8: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
5217correctmath.counterfactual.base-v1conf 100% · 339ms · $0.005 · 348 tok
question
Work strictly in base 11. Multiply the base-11 numbers 21 and 57. Give the result IN BASE 11 (digits beyond 9 are A, B, C). End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1087correctmath.algebra.system-v2conf 100% · 361ms · $0.006 · 479 tok
question
Solve the system, then answer the derived question. 2x + 2y = 90 3x − 7y = -145 What is the value of 2x − 4y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-78correctmath.percent.chain-v2conf 100% · 304ms · $0.004 · 262 tok
question
An inventory starts at 76000 units. The delivery van has a 67-liter fuel tank. In the first month the inventory grows by 36%. The delivery van has a 83-liter fuel tank. The next month it shrinks by 44%, and the month after it grows by 30%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
75246.08correctmath.arith.chain-v2conf 100% · 686ms · $0.005 · 356 tok
question
Compute the value of the following expression. (((82 × 87 − 481) × 5 + 7150) − 62 × 67) × 7 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
253827correctmath.chained.pipeline-v1conf 100% · 477ms · $0.004 · 293 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 70 × 63. Step 2: Q = P × 5 − 926. Step 3: divide Q by 6: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
3524wrongmath.counterfactual.base-v1anchorconf 100% · 743ms · $0.004 · 317 tok
model answer:
11233correctmath.percent.chain-v2anchorconf 100% · 305ms · $0.004 · 301 tok
model answer:
61,896.52correctmath.algebra.system-v2anchorconf 100% · 438ms · $0.005 · 365 tok
model answer:
87correctmath.arith.chain-v2anchorconf 100% · 438ms · $0.005 · 348 tok
model answer:
108153multilingual 28/30 correct
correctmultilingual.wordnum-v1conf 100% · 707ms · $0.003 · 216 tok
question
A number is written in French: « trois cent quatre-vingt-quatorze ». Another is written in Spanish: « seiscientos setenta y nueve ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1073correctmultilingual.numword-v2conf 100% · 1.1s · $0.002 · 134 tok
question
Compute 56 + 173, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
deux cent vingt-neufcorrectmultilingual.wordnum-v1conf 100% · 618ms · $0.002 · 165 tok
question
A number is written in French: « trois cent quarante et un ». Another is written in Spanish: « sesenta y nueve ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
272correctmultilingual.numword-v2conf 100% · 286ms · $0.002 · 97 tok
question
Compute 130 + 428, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quinientos cincuenta y ochowrongmultilingual.wordnum-v1conf 100% · 290ms · $0.002 · 109 tok
question
A number is written in French: « deux cent soixante-quatorze ». Another is written in Spanish: « ciento sesenta y ocho ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
432correctmultilingual.wordnum-v1conf 100% · 355ms · $0.003 · 196 tok
question
A number is written in French: « huit cent quarante-neuf ». Another is written in Spanish: « doscientos diez ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1059correctmultilingual.numword-v2conf 100% · 771ms · $0.001 · 85 tok
question
Compute 492 + 124, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
seiscientos dieciséiscorrectmultilingual.numword-v2conf 100% · 267ms · $0.002 · 128 tok
question
Compute 118 + 150, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
deux cent soixante-huitwrongmultilingual.wordnum-v1conf 100% · 287ms · $0.002 · 107 tok
question
A number is written in French: « trois cent soixante-douze ». Another is written in Spanish: « quinientos ochenta y uno ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-219correctmultilingual.numword-v2conf 100% · 315ms · $0.002 · 132 tok
question
Compute 307 + 305, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
seiscientos docecorrectmultilingual.numword-v2conf 100% · 789ms · $0.003 · 198 tok
question
Compute 136 + 135, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
doscientos setenta y unocorrectmultilingual.wordnum-v1conf 100% · 268ms · $0.002 · 103 tok
question
A number is written in French: « trois cent soixante-sept ». Another is written in Spanish: « ochocientos setenta ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-503correctmultilingual.wordnum-v1conf 100% · 272ms · $0.002 · 168 tok
question
A number is written in French: « cinq cent vingt-trois ». Another is written in Spanish: « setecientos ochenta y cinco ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1308correctmultilingual.numword-v2conf 100% · 505ms · $0.002 · 153 tok
question
Compute 299 + 98, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
trois cent quatre-vingt-dix-septcorrectmultilingual.wordnum-v1conf 100% · 314ms · $0.002 · 153 tok
question
A number is written in French: « sept cent trente ». Another is written in Spanish: « quinientos ochenta y seis ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
144correctmultilingual.wordnum-v1conf 100% · 337ms · $0.003 · 186 tok
question
A number is written in French: « cinq cent quatre-vingt-huit ». Another is written in Spanish: « ochocientos dos ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-214correctmultilingual.numword-v2conf 100% · 283ms · $0.002 · 136 tok
question
Compute 478 + 96, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quinientos setenta y cuatrocorrectmultilingual.numword-v2conf 100% · 262ms · $0.002 · 132 tok
question
Compute 131 + 335, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quatre cent soixante-sixcorrectmultilingual.wordnum-v1conf 100% · 343ms · $0.002 · 124 tok
question
A number is written in French: « soixante-seize ». Another is written in Spanish: « ciento ochenta y tres ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-107correctmultilingual.wordnum-v1conf 100% · 325ms · $0.003 · 179 tok
question
A number is written in French: « six cent quarante-trois ». Another is written in Spanish: « seiscientos noventa y dos ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-49correctmultilingual.numword-v2conf 100% · 286ms · $0.002 · 122 tok
question
Compute 465 + 388, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
huit cent cinquante-troiscorrectmultilingual.wordnum-v1conf 100% · 292ms · $0.003 · 188 tok
question
A number is written in French: « neuf cent quatre-vingts ». Another is written in Spanish: « trescientos diecinueve ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1299correctmultilingual.numword-v2conf 100% · 273ms · $0.002 · 99 tok
question
Compute 306 + 297, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
six cent troiscorrectmultilingual.wordnum-v1conf 100% · 318ms · $0.003 · 213 tok
question
A number is written in French: « soixante-dix-huit ». Another is written in Spanish: « quinientos ochenta y uno ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-503correctmultilingual.numword-v2conf 100% · 538ms · $0.002 · 130 tok
question
Compute 208 + 273, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quatre cent quatre-vingt-uncorrectmultilingual.numword-v2conf 100% · 232ms · $0.003 · 196 tok
question
Compute 131 + 394, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
cinq cent vingt-cinqcorrectmultilingual.wordnum-v1anchorconf 100% · 309ms · $0.003 · 176 tok
model answer:
150correctmultilingual.wordnum-v1anchorconf 100% · 443ms · $0.003 · 249 tok
model answer:
762correctmultilingual.numword-v2anchorconf 100% · 334ms · $0.002 · 151 tok
model answer:
huit cent soixante-dix-neufcorrectmultilingual.numword-v2anchorconf 100% · 288ms · $0.002 · 122 tok
model answer:
seiscientos ochoreasoning 25/30 correct
wrongreasoning.deduction.order-v2conf 95% · 283ms · $0.003 · 186 tok
question
Seven people are ranked by who is taller (rank 1 = tallest). Alice is taller than Hana. Jonas is taller than Hana. Alice is taller than Jonas. Bruno is taller than Emil. Emil is taller than Alice. Emil is taller than Jonas. Chen is heavier than everyone here, but Chen is not being ranked. Mona is taller than Dara. Bruno is taller than Alice. Dara is taller than Bruno. Who is second (rank 2)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brunocorrectreasoning.deduction.position-v1conf 100% · 290ms · $0.002 · 154 tok
question
Four people stand in a queue (number 1 is the front). Ines is directly ahead of Tessa. Goran is directly ahead of Liam. Tessa is number 4 in the queue. Liam is directly ahead of Ines. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Inescorrectreasoning.deduction.order-v2conf 100% · 288ms · $0.004 · 284 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Mona is heavier than Nadir. Kira is heavier than Rosa. Quinn is heavier than Liam. Ines is heavier than Rosa. Nadir is heavier than Kira. Jonas is taller than everyone here, but Jonas is not being ranked. Liam is heavier than Mona. Mona is heavier than Kira. Ines is heavier than Nadir. Mona is heavier than Ines. Who is second (rank 2)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Liamcorrectreasoning.deduction.position-v1conf 100% · 303ms · $0.002 · 133 tok
question
Four people stand in a queue (number 1 is the front). Sami is number 4 in the queue. Quinn is directly ahead of Sami. Ola is directly ahead of Quinn. Tessa is directly ahead of Ola. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Quinncorrectreasoning.deduction.order-v2conf 100% · 308ms · $0.004 · 269 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Bruno is faster than Priya. Sami is faster than Emil. Sami is faster than Bruno. Rosa is faster than Bruno. Emil is faster than Rosa. Rosa is faster than Kira. Quinn is faster than Kira. Bruno is faster than Quinn. Priya is faster than Quinn. Mona is taller than everyone here, but Mona is not being ranked. Who is fifth (rank 5)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Priyacorrectreasoning.deduction.position-v1conf 100% · 249ms · $0.002 · 124 tok
question
Four people stand in a queue (number 1 is the front). Nadir is directly ahead of Hana. Emil is number 3 in the queue. Hana is directly ahead of Emil. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Emilcorrectreasoning.deduction.order-v2conf 100% · 329ms · $0.004 · 307 tok
question
Seven people are ranked by who is taller (rank 1 = tallest). Ines is heavier than everyone here, but Ines is not being ranked. Hana is taller than Ola. Mona is taller than Dara. Ola is taller than Mona. Dara is taller than Sami. Mona is taller than Liam. Sami is taller than Liam. Liam is taller than Kira. Sami is taller than Kira. Mona is taller than Kira. Who is fifth (rank 5)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Samicorrectreasoning.deduction.position-v1conf 100% · 370ms · $0.002 · 156 tok
question
Four people stand in a queue (number 1 is the front). Ola is number 2 in the queue. Quinn is directly ahead of Liam. Rosa is directly ahead of Ola. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Liamcorrectreasoning.deduction.position-v1conf 100% · 336ms · $0.002 · 133 tok
question
Four people stand in a queue (number 1 is the front). Sami is directly ahead of Dara. Farah is directly ahead of Rosa. Rosa is number 2 in the queue. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Samicorrectreasoning.deduction.order-v2conf 95% · 340ms · $0.007 · 485 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Ines is faster than Emil. Dara is faster than Ines. Nadir is faster than Mona. Ola is older than everyone here, but Ola is not being ranked. Mona is faster than Priya. Dara is faster than Emil. Priya is faster than Farah. Nadir is faster than Dara. Farah is faster than Dara. Priya is faster than Dara. Who is fifth (rank 5)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Daracorrectreasoning.deduction.order-v2conf 100% · 495ms · $0.004 · 245 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Jonas is heavier than everyone here, but Jonas is not being ranked. Tessa is older than Emil. Hana is older than Bruno. Bruno is older than Emil. Tessa is older than Chen. Bruno is older than Tessa. Chen is older than Nadir. Bruno is older than Chen. Nadir is older than Emil. Ola is older than Hana. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brunocorrectreasoning.deduction.position-v1conf 100% · 301ms · $0.003 · 216 tok
question
Four people stand in a queue (number 1 is the front). Goran is directly ahead of Dara. Liam is number 1 in the queue. Dara is directly ahead of Quinn. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Quinncorrectreasoning.deduction.position-v1conf 100% · 360ms · $0.001 · 80 tok
question
Four people stand in a queue (number 1 is the front). Bruno is directly ahead of Quinn. Jonas is directly ahead of Bruno. Emil is number 1 in the queue. Who is number 2? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Jonascorrectreasoning.deduction.order-v2conf 100% · 308ms · $0.003 · 183 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Priya is heavier than Goran. Rosa is heavier than Priya. Liam is heavier than Alice. Liam is heavier than Ines. Rosa is heavier than Liam. Goran is heavier than Liam. Tessa is heavier than Alice. Alice is heavier than Ines. Bruno is taller than everyone here, but Bruno is not being ranked. Liam is heavier than Tessa. Who is fourth (rank 4)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Liamcorrectreasoning.deduction.position-v1conf 100% · 321ms · $0.002 · 141 tok
question
Four people stand in a queue (number 1 is the front). Mona is directly ahead of Goran. Sami is directly ahead of Ines. Goran is directly ahead of Sami. Ines is number 4 in the queue. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Samiwrongreasoning.deduction.order-v2conf 100% · 281ms · $0.003 · 226 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Alice is heavier than Quinn. Tessa is heavier than Dara. Alice is heavier than Rosa. Rosa is heavier than Dara. Hana is taller than everyone here, but Hana is not being ranked. Rosa is heavier than Tessa. Chen is heavier than Tessa. Rosa is heavier than Chen. Quinn is heavier than Rosa. Ola is heavier than Alice. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Rosawrongreasoning.deduction.position-v1conf 100% · 306ms · $0.004 · 323 tok
question
Four people stand in a queue (number 1 is the front). Alice is directly ahead of Goran. Tessa is number 3 in the queue. Goran is directly ahead of Tessa. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
The fourth personcorrectreasoning.deduction.order-v2conf 100% · 313ms · $0.003 · 174 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Ola is heavier than Sami. Ola is heavier than Mona. Bruno is heavier than Alice. Farah is older than everyone here, but Farah is not being ranked. Mona is heavier than Sami. Alice is heavier than Ola. Alice is heavier than Mona. Sami is heavier than Quinn. Quinn is heavier than Kira. Ola is heavier than Sami. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olacorrectreasoning.deduction.order-v2conf 100% · 350ms · $0.003 · 193 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Priya is faster than Sami. Tessa is faster than Priya. Sami is faster than Ola. Nadir is older than everyone here, but Nadir is not being ranked. Kira is faster than Alice. Kira is faster than Tessa. Hana is faster than Alice. Alice is faster than Priya. Tessa is faster than Hana. Priya is faster than Ola. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Hanacorrectreasoning.deduction.position-v1conf 100% · 346ms · $0.002 · 107 tok
question
Four people stand in a queue (number 1 is the front). Hana is directly ahead of Sami. Farah is directly ahead of Hana. Sami is number 3 in the queue. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Farahcorrectreasoning.deduction.position-v1conf 100% · 299ms · $0.002 · 154 tok
question
Four people stand in a queue (number 1 is the front). Quinn is directly ahead of Bruno. Rosa is number 3 in the queue. Bruno is directly ahead of Rosa. Who is number 2? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brunocorrectreasoning.deduction.order-v2conf 100% · 326ms · $0.003 · 177 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Mona is older than Sami. Quinn is older than Goran. Sami is older than Quinn. Dara is older than Goran. Quinn is older than Goran. Farah is heavier than everyone here, but Farah is not being ranked. Chen is older than Mona. Quinn is older than Dara. Ines is older than Chen. Sami is older than Dara. Who is sixth (rank 6)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Daracorrectreasoning.deduction.position-v1conf 100% · 310ms · $0.002 · 110 tok
question
Four people stand in a queue (number 1 is the front). Rosa is directly ahead of Kira. Priya is directly ahead of Rosa. Kira is directly ahead of Mona. Mona is number 4 in the queue. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Priyacorrectreasoning.deduction.order-v2conf 100% · 331ms · $0.003 · 232 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Mona is faster than Dara. Kira is faster than Nadir. Ola is faster than Kira. Kira is faster than Emil. Goran is faster than Ola. Emil is faster than Nadir. Ola is faster than Emil. Quinn is taller than everyone here, but Quinn is not being ranked. Dara is faster than Goran. Goran is faster than Kira. Who is fourth (rank 4)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olawrongreasoning.deduction.order-v2conf 95% · 274ms · $0.005 · 343 tok
question
Seven people are ranked by who is taller (rank 1 = tallest). Liam is taller than Nadir. Jonas is heavier than everyone here, but Jonas is not being ranked. Chen is taller than Priya. Chen is taller than Nadir. Hana is taller than Chen. Sami is taller than Liam. Priya is taller than Rosa. Chen is taller than Liam. Sami is taller than Nadir. Rosa is taller than Sami. Who is fourth (rank 4)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Liamcorrectreasoning.deduction.position-v1conf 100% · 339ms · $0.002 · 101 tok
question
Four people stand in a queue (number 1 is the front). Alice is number 3 in the queue. Dara is directly ahead of Alice. Farah is directly ahead of Dara. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Alicecorrectreasoning.deduction.position-v1anchorconf 100% · 277ms · $0.003 · 194 tok
model answer:
Quinncorrectreasoning.deduction.order-v2anchorconf 100% · 506ms · $0.004 · 251 tok
model answer:
Quinnwrongreasoning.deduction.order-v2anchorconf 100% · 319ms · $0.004 · 311 tok
model answer:
Alicecorrectreasoning.deduction.position-v1anchorconf 100% · 542ms · $0.002 · 147 tok
model answer:
FarahRun history
- 2026-08-05v0.2.0index_fit777
- 2026-08-05v0.1.0index_fit712
- 2026-08-05v0.1.0index_fit696