← Leaderboard
Anthropic: Claude Opus 4.8
anthropic/claude-opus-4.8 · anthropic · context 1 000 000 · in $5.00/1M · out $25.00/1M
Global Index
796
95% CI [748–843] · index v0.2.0
Per-domain scores
| Domain | Score (95% CI) | Accuracy (IRT) | Consistency | Calibration | Contam. Δ | p50 | $/1k | |
|---|---|---|---|---|---|---|---|---|
| agentic | 844 [745–943] | 0.807 | 0.97 | 0.94 | 0.038 | 1.3s | $18.93 | |
| code | 874 [752–996] | 0.791 | 1.00 | 0.99 | 0.000 | 1.3s | $12.92 | |
| instruction following | 812 [678–947] | 0.731 | 0.83 | 0.99 | 0.000 | 1.7s | $4.83 | |
| knowledge | 729 [556–902] | 0.549 | 1.00 | 0.99 | 0.000 | 1.3s | $1.15 | |
| math | 836 [679–992] | 0.728 | 1.00 | 0.99 | 0.000 | 1.3s | $5.99 | |
| multilingual | 821 [658–983] | 0.706 | 1.00 | 0.98 | 0.000 | 1.1s | $2.26 | |
| reasoning | 722 [573–872] | 0.651 | 0.97 | 0.95 | 0.077 | 1.3s | $6.57 | |
| terminal | 848 [748–949] | 0.803 | 0.98 | 0.98 | 0.038 | 1.3s | $9.83 | |
| vision ocr | 676 [514–837] | 0.513 | 1.00 | 0.97 | 0.038 | 1.8s | $5.54 |
Every answer, every test
Full transparency: the model's latest graded answer on every instantiated item of the current index version. Answer keys never leave the server; anchor-item prompts are withheld to protect the longitudinal subset.
agentic 29/30 correct
correctagentic.tools.ledger-v1conf 99% · 5.0s · $0.020 · 676 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- lima: $299
- delta: $199
- kilo: $631
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $221 from "lima" to "delta"
2. pay $183 from "lima" to "delta"
3. pay $155 from "lima" to "kilo"
4. pay $482 from "lima" to "kilo"
5. pay $538 from "lima" to "delta"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "lima", "to": "delta", "amount": 221}},
{"tool": "top_up_from_reserve", "args": {"account": "lima", "amount": 105}},
{"tool": "transfer", "args": {"from": "lima", "to": "delta", "amount": 183}},
{"tool": "top_up_from_reserve", "args": {"account": "lima", "amount": 155}},
{"tool": "transfer", "args": {"from": "lima", "to": "kilo", "amount": 155}},
{"tool": "top_up_from_reserve", "args": {"account": "lima", "amount": 482}},
{"tool": "transfer", "args": {"from": "lima", "to": "kilo", "amount": 482}},
{"tool": "top_up_from_reserve", "args": {"account": "lima", "amount": 538}},
{"tool": "transfer", "args": {"from": "lima", "to": "delta", "amount": 538}}
]correctagentic.tools.context-load-v1conf 97% · 2.6s · $0.038 · 570 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (249 records, format: id|customer|region|item|qty|status):
```
1810|dorian|north|pump|37|shipped
1770|ionic|south|panel|96|shipped
1593|birch|east|valve|88|held
1890|ember|south|gasket|45|held
2140|ember|north|panel|16|held
1666|ember|north|pump|14|paid
1854|gale|north|panel|51|paid
1913|cobalt|east|frame|65|pending
1881|dorian|east|sensor|94|held
2174|dorian|west|cable|43|shipped
2312|ionic|south|panel|26|shipped
1655|birch|west|pump|10|pending
2250|birch|west|cable|93|shipped
1518|cobalt|north|rotor|25|held
2425|gale|south|valve|16|held
2278|gale|north|sensor|16|pending
2056|ionic|south|rotor|60|held
2073|cobalt|south|frame|64|shipped
2287|gale|east|cable|25|held
1929|cobalt|north|frame|32|paid
2191|fulton|south|cable|74|shipped
1700|dorian|east|gasket|53|held
1849|birch|west|valve|68|held
2261|juno|east|cable|63|paid
1940|acme|south|cable|76|pending
2118|ionic|east|rotor|59|pending
1552|birch|west|valve|48|pending
2295|harbor|south|valve|26|held
2189|fulton|north|gasket|32|held
1821|juno|north|panel|30|shipped
2079|gale|south|cable|12|held
1522|cobalt|north|rotor|14|pending
2025|fulton|east|sensor|99|held
2042|ember|north|pump|74|paid
2479|ionic|west|panel|89|pending
2284|harbor|north|valve|57|shipped
1614|birch|east|frame|24|paid
2014|gale|east|pump|31|pending
1585|fulton|south|sensor|87|paid
1672|cobalt|south|sensor|73|held
2108|cobalt|west|frame|44|paid
2149|ionic|north|panel|15|pending
1935|dorian|east|valve|96|pending
1921|birch|south|gasket|83|shipped
2054|acme|west|cable|32|held
2020|juno|west|gasket|82|paid
1683|birch|west|rotor|14|pending
1630|ember|south|cable|93|pending
2390|birch|east|gasket|24|paid
1678|gale|north|valve|31|paid
2352|gale|south|gasket|69|held
2080|acme|north|valve|19|pending
1660|ionic|east|valve|79|paid
2321|juno|north|rotor|48|held
2432|ember|south|pump|65|pending
2213|dorian|south|frame|20|held
2362|birch|south|cable|88|pending
1744|gale|west|pump|25|shipped
1980|harbor|south|sensor|54|paid
1648|harbor|west|valve|35|shipped
1681|birch|east|pump|36|held
1645|harbor|west|pump|79|held
2442|fulton|north|rotor|54|paid
1530|cobalt|north|gasket|72|paid
1996|acme|east|gasket|40|shipped
1548|fulton|west|gasket|17|paid
1567|birch|south|panel|11|held
2454|juno|north|gasket|10|pending
1727|juno|east|pump|60|paid
1587|dorian|south|cable|43|held
2022|gale|east|rotor|24|held
2383|dorian|north|gasket|99|paid
1766|harbor|east|valve|79|paid
1488|cobalt|north|rotor|52|pending
2238|dorian|north|cable|43|pending
2302|juno|south|sensor|64|shipped
2462|acme|east|frame|47|held
1509|cobalt|west|cable|28|pending
1785|gale|north|gasket|70|pending
2254|ember|east|valve|81|held
2482|acme|east|frame|29|paid
1728|cobalt|east|frame|86|held
2293|juno|north|panel|66|pending
1677|gale|west|valve|81|pending
1497|cobalt|north|panel|79|paid
1952|fulton|north|frame|71|paid
1555|birch|east|panel|77|paid
2039|cobalt|north|pump|77|shipped
2068|gale|north|rotor|53|shipped
1908|ionic|south|valve|97|pending
1493|cobalt|east|frame|15|pending
2315|harbor|east|frame|67|paid
2345|dorian|south|gasket|35|shipped
1860|ionic|north|rotor|15|pending
1969|juno|south|gasket|34|pending
1592|acme|west|sensor|60|held
1835|cobalt|south|panel|28|shipped
2368|fulton|south|rotor|98|paid
2002|fulton|north|valve|56|shipped
2459|acme|south|gasket|77|pending
1845|birch|south|frame|84|shipped
1502|cobalt|north|gasket|62|pending
1864|cobalt|north|cable|85|paid
1707|juno|south|cable|36|pending
2053|birch|west|cable|79|shipped
1777|cobalt|south|sensor|61|pending
2160|dorian|west|panel|88|pending
1926|ember|east|cable|62|pending
1578|juno|north|cable|82|shipped
1537|cobalt|north|frame|73|pending
1619|fulton|east|frame|50|paid
2308|gale|west|sensor|74|held
2161|ember|south|gasket|74|pending
1726|juno|south|gasket|97|held
1720|fulton|east|cable|66|paid
1601|dorian|west|gasket|15|held
2170|birch|south|valve|76|held
1517|cobalt|south|sensor|61|pending
1967|dorian|south|valve|67|paid
2133|birch|west|rotor|78|pending
2036|ionic|east|valve|13|pending
2198|harbor|west|rotor|81|shipped
1804|fulton|south|valve|96|shipped
1724|birch|east|rotor|82|paid
2418|ember|north|cable|80|paid
2410|gale|south|rotor|46|held
1971|cobalt|west|rotor|23|pending
1948|dorian|south|frame|31|shipped
2215|dorian|east|pump|38|pending
2165|birch|east|pump|60|paid
1547|cobalt|north|valve|22|shipped
2360|dorian|north|cable|43|shipped
2051|harbor|north|cable|31|held
2438|juno|north|rotor|42|shipped
2226|birch|east|panel|81|pending
1633|ionic|south|frame|83|held
1960|acme|east|cable|68|paid
1796|ember|north|panel|26|held
2370|gale|east|gasket|30|shipped
1512|cobalt|north|sensor|89|paid
1838|harbor|east|rotor|30|paid
2272|ember|west|rotor|32|held
2030|ember|north|gasket|31|shipped
1595|cobalt|north|panel|22|held
2090|fulton|north|panel|31|paid
1781|dorian|south|rotor|62|held
2274|harbor|east|panel|88|pending
1824|fulton|south|rotor|41|shipped
1916|birch|east|rotor|24|shipped
2062|acme|west|valve|47|pending
1992|juno|south|valve|10|pending
2142|ember|south|sensor|56|pending
1991|ionic|west|cable|42|paid
2326|harbor|south|panel|42|shipped
2180|ionic|north|frame|82|pending
2350|cobalt|west|gasket|29|pending
1625|ionic|north|pump|45|paid
1639|gale|south|valve|15|pending
2414|juno|south|cable|26|paid
2347|birch|west|frame|56|held
1631|dorian|south|sensor|58|paid
1904|acme|west|valve|67|pending
1525|cobalt|south|valve|92|pending
1715|juno|east|sensor|38|held
2175|juno|south|cable|75|shipped
1974|ionic|east|valve|80|shipped
2369|harbor|north|pump|25|shipped
2183|acme|east|panel|11|paid
1828|ionic|east|frame|68|shipped
1898|ember|east|frame|57|paid
2476|harbor|north|rotor|76|pending
2282|ionic|south|sensor|24|shipped
2435|gale|south|frame|88|paid
1675|ionic|east|frame|59|held
1562|juno|east|pump|86|held
1906|harbor|north|frame|38|held
2396|juno|west|pump|73|held
2231|ionic|west|valve|42|paid
2290|birch|south|frame|85|shipped
1955|cobalt|east|rotor|82|shipped
2235|juno|south|gasket|55|shipped
2339|juno|west|sensor|60|pending
2412|harbor|north|panel|37|held
1774|ember|north|pump|77|held
2354|fulton|north|gasket|45|held
2009|harbor|north|sensor|13|shipped
2123|acme|south|valve|28|shipped
1993|fulton|east|gasket|42|held
2206|juno|north|panel|58|held
1771|fulton|west|rotor|45|pending
2221|dorian|east|panel|40|paid
1817|cobalt|south|panel|28|held
2292|fulton|south|rotor|58|shipped
2467|harbor|south|gasket|98|paid
1513|cobalt|north|sensor|94|pending
1944|ember|east|sensor|50|shipped
2031|fulton|east|frame|59|pending
2204|cobalt|north|frame|75|paid
1800|gale|south|panel|99|held
2333|birch|east|panel|88|shipped
2264|acme|east|gasket|78|held
2451|birch|north|panel|27|pending
2083|ionic|north|rotor|28|held
1876|gale|south|frame|90|pending
1714|ember|north|panel|45|shipped
2227|cobalt|west|cable|22|paid
1694|cobalt|south|pump|18|held
2473|fulton|west|gasket|64|paid
2034|fulton|north|gasket|64|held
2115|dorian|south|cable|36|shipped
2386|fulton|east|gasket|92|held
1893|gale|west|pump|83|shipped
1737|ember|east|panel|58|held
1750|juno|south|valve|89|pending
1735|acme|west|panel|31|paid
1579|birch|north|cable|18|paid
1754|ember|east|gasket|87|shipped
1736|ionic|south|panel|59|paid
2047|harbor|east|cable|72|held
1574|acme|west|valve|20|paid
1605|fulton|east|valve|39|held
1741|ember|south|sensor|27|held
1869|fulton|south|valve|59|shipped
1759|gale|north|rotor|61|shipped
2377|ionic|north|valve|89|paid
1966|dorian|south|cable|45|paid
2268|fulton|north|frame|18|paid
2409|cobalt|east|sensor|62|paid
2244|ember|north|cable|23|held
1688|harbor|west|sensor|63|pending
1795|cobalt|east|pump|49|paid
2052|harbor|east|sensor|20|paid
2130|acme|north|frame|89|held
2403|harbor|east|rotor|84|shipped
2447|juno|west|gasket|97|paid
1757|harbor|north|gasket|38|pending
1984|ionic|west|frame|18|pending
1844|fulton|north|gasket|44|held
1887|dorian|north|pump|17|paid
2103|juno|west|gasket|75|shipped
1820|ionic|west|panel|54|shipped
2398|ember|west|valve|30|paid
1877|birch|north|panel|30|shipped
2097|fulton|north|cable|26|held
1609|harbor|north|pump|83|held
1544|cobalt|east|valve|66|pending
2156|fulton|west|pump|59|held
2437|fulton|north|cable|63|pending
1790|ionic|east|panel|20|shipped
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "cobalt" AND region = "north" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 49, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "rotor", "qty": 52}},
{"tool": "restock", "args": {"item": "gasket", "qty": 62}},
{"tool": "restock", "args": {"item": "sensor", "qty": 94}},
{"tool": "cancel_order", "args": {"order_id": 1522}},
{"tool": "restock", "args": {"item": "frame", "qty": 73}}
]correctagentic.tools.triage-v1conf 96% · 1.1s · $0.010 · 265 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 7, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → rivera
- data → haddad
- infra → okafor
INCIDENTS:
1. "refund double-charged" (category: payments, priority 5)
2. "refund double-charged" (category: payments, priority 5)
3. "webhooks not delivered" (category: infra, priority 6)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "rivera"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 5}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-2", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "webhooks not delivered", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "okafor"}}
]correctagentic.tools.deploy-v1conf 96% · 1.1s · $0.011 · 339 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- notifier: billing, gateway
- billing: (none)
- gateway: billing
- auth-svc: notifier
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "notifier" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "billing"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "health_check", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "auth-svc"}}
]correctagentic.tools.triage-v1conf 96% · 7.9s · $0.012 · 344 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 8, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- infra → silva
- data → dubois
- payments → chen
INCIDENTS:
1. "uploads failing intermittently" (category: infra, priority 6)
2. "uploads failing intermittently" (category: infra, priority 6)
3. "invoice total wrong" (category: payments, priority 7)
4. "API latency spikes" (category: infra, priority 6)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "uploads failing intermittently", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "uploads failing intermittently", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-2", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "invoice total wrong", "priority": 7}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "chen"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "silva"}}
]correctagentic.tools.ledger-v1conf 99% · 1.2s · $0.015 · 462 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- oscar: $683
- bravo: $297
- lima: $422
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $388 from "lima" to "oscar"
2. pay $127 from "oscar" to "bravo"
3. pay $307 from "bravo" to "oscar"
4. pay $407 from "bravo" to "lima"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "lima", "to": "oscar", "amount": 388}},
{"tool": "transfer", "args": {"from": "oscar", "to": "bravo", "amount": 127}},
{"tool": "transfer", "args": {"from": "bravo", "to": "oscar", "amount": 307}},
{"tool": "top_up_from_reserve", "args": {"account": "bravo", "amount": 290}},
{"tool": "transfer", "args": {"from": "bravo", "to": "lima", "amount": 407}}
]correctagentic.tools.context-load-v1conf 98% · 1.5s · $0.033 · 589 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (193 records, format: id|customer|region|item|qty|status):
```
1921|birch|south|rotor|85|paid
1497|ember|north|sensor|21|held
2066|juno|west|panel|71|shipped
1788|fulton|west|panel|60|shipped
2153|juno|east|gasket|61|pending
1800|fulton|west|valve|79|shipped
1459|juno|east|pump|36|pending
2050|cobalt|east|rotor|12|pending
1705|harbor|west|frame|79|shipped
1738|ember|west|sensor|97|held
1693|gale|south|pump|36|shipped
1762|acme|north|sensor|38|held
1545|acme|north|sensor|85|pending
2136|harbor|east|panel|43|shipped
2002|gale|north|rotor|21|pending
1959|acme|north|sensor|56|pending
2149|gale|west|pump|85|held
1496|fulton|east|cable|51|shipped
1482|cobalt|east|pump|44|shipped
2231|acme|west|panel|98|paid
1764|cobalt|south|panel|47|shipped
2029|acme|east|sensor|65|paid
1740|acme|east|sensor|76|pending
1890|fulton|east|pump|33|pending
1795|acme|west|sensor|57|shipped
1581|dorian|east|cable|52|held
1563|ember|west|pump|99|shipped
2160|ionic|east|valve|89|held
1472|juno|east|panel|55|pending
1504|juno|east|panel|83|held
1622|gale|south|frame|67|paid
2210|ember|north|gasket|82|held
1509|fulton|west|gasket|60|shipped
1895|acme|north|gasket|39|pending
2011|ember|west|panel|95|paid
2197|acme|south|panel|23|held
1668|juno|east|frame|94|held
1687|dorian|west|valve|56|paid
1700|harbor|north|cable|98|pending
2220|fulton|south|frame|13|shipped
1857|birch|west|sensor|70|held
2097|ionic|east|cable|35|held
1884|birch|west|frame|70|pending
1532|dorian|south|panel|48|pending
2007|harbor|west|pump|12|held
2185|birch|north|sensor|50|pending
2037|gale|east|rotor|24|held
1941|ionic|south|sensor|53|paid
2016|juno|south|cable|90|paid
2181|ionic|west|sensor|38|held
1811|acme|west|panel|43|paid
1797|cobalt|east|sensor|32|pending
1626|gale|west|pump|94|paid
1929|ember|south|rotor|95|paid
1751|birch|north|sensor|16|pending
2055|acme|east|valve|81|held
1726|gale|east|frame|77|shipped
1848|birch|north|sensor|78|pending
1648|birch|south|panel|43|pending
2040|fulton|east|valve|86|held
1511|cobalt|west|rotor|70|held
1877|dorian|south|frame|26|held
2177|cobalt|north|gasket|49|pending
1458|juno|east|panel|92|shipped
1935|juno|south|cable|26|paid
1577|juno|west|cable|60|pending
1742|juno|west|gasket|27|shipped
1679|dorian|west|pump|58|pending
1951|cobalt|west|sensor|96|held
2170|birch|south|frame|24|pending
2172|fulton|south|gasket|21|shipped
1729|gale|west|rotor|61|shipped
2227|juno|south|valve|82|paid
1975|fulton|west|cable|88|paid
1820|acme|north|sensor|26|pending
2200|dorian|west|frame|76|pending
1943|cobalt|east|rotor|50|paid
1680|ionic|east|cable|45|pending
2109|acme|east|cable|81|pending
2142|birch|north|frame|83|shipped
1993|ember|north|rotor|40|held
1598|acme|north|gasket|23|paid
1638|ionic|west|sensor|17|pending
1781|dorian|east|pump|29|paid
1716|ionic|south|valve|38|pending
1642|ember|south|valve|38|pending
1771|juno|east|pump|82|shipped
1813|birch|east|gasket|28|pending
1614|cobalt|south|pump|34|paid
1756|ember|west|rotor|83|held
1810|ember|south|rotor|52|shipped
1604|ember|south|rotor|44|paid
1855|acme|west|sensor|97|pending
1455|juno|north|cable|50|pending
2024|ember|north|gasket|13|paid
2030|ionic|north|gasket|42|shipped
2048|birch|east|gasket|57|held
2190|fulton|south|gasket|75|held
1834|cobalt|east|frame|46|held
1530|gale|south|valve|90|paid
1625|gale|south|pump|83|held
1684|ember|west|cable|19|paid
1902|harbor|north|panel|25|pending
1761|acme|east|rotor|26|paid
1560|juno|east|gasket|80|paid
2079|birch|east|rotor|11|paid
2045|birch|north|panel|59|shipped
1957|harbor|south|frame|25|pending
1659|gale|north|gasket|61|shipped
1945|harbor|south|valve|36|paid
1988|birch|south|valve|30|pending
2237|acme|west|pump|30|held
2130|cobalt|north|valve|61|shipped
2213|ionic|west|cable|90|held
2217|juno|north|valve|33|held
1896|ember|north|cable|55|pending
1569|acme|east|rotor|39|paid
1864|fulton|north|pump|35|pending
1808|fulton|north|panel|98|paid
1652|birch|north|gasket|46|shipped
2208|fulton|west|pump|31|shipped
1871|harbor|west|panel|50|shipped
1982|ionic|south|cable|73|held
2028|harbor|east|frame|53|held
1607|fulton|north|valve|58|held
1550|dorian|south|pump|84|held
1969|juno|west|panel|66|held
1746|ionic|east|gasket|94|shipped
1674|dorian|north|panel|97|pending
1798|juno|west|pump|64|pending
1828|fulton|south|cable|81|pending
1947|birch|south|sensor|42|held
2086|acme|east|rotor|23|paid
2067|harbor|east|valve|36|paid
1517|juno|west|panel|43|pending
2091|birch|north|sensor|13|paid
1841|gale|north|pump|85|paid
1523|dorian|west|valve|22|paid
2113|dorian|south|cable|34|shipped
1503|harbor|east|sensor|74|held
2076|fulton|south|panel|14|paid
1573|juno|east|sensor|87|held
2099|juno|west|pump|29|shipped
1909|ember|west|cable|35|paid
1714|acme|west|rotor|74|shipped
1453|juno|east|rotor|30|pending
2154|acme|north|frame|56|shipped
1477|juno|south|rotor|40|pending
1727|cobalt|west|rotor|32|pending
1914|ionic|west|pump|87|shipped
1616|birch|north|frame|43|paid
2059|harbor|east|panel|80|held
2071|juno|east|valve|40|shipped
2165|fulton|east|rotor|40|paid
1926|dorian|east|panel|64|held
1805|gale|west|panel|21|pending
1465|juno|south|gasket|69|pending
1630|harbor|east|pump|55|held
2202|harbor|north|panel|31|pending
1478|juno|east|cable|56|shipped
1719|birch|north|rotor|46|held
2134|ionic|west|gasket|19|shipped
2126|ionic|west|frame|64|held
1736|dorian|east|pump|70|paid
1466|juno|east|gasket|33|paid
1655|dorian|west|sensor|50|pending
2105|harbor|north|rotor|93|paid
1965|dorian|west|valve|93|shipped
1744|cobalt|east|rotor|69|paid
1632|cobalt|east|frame|49|paid
1489|juno|west|sensor|90|pending
1752|gale|west|panel|28|held
1999|juno|west|sensor|34|held
2140|birch|north|pump|16|shipped
1552|dorian|north|frame|21|held
1741|cobalt|south|sensor|85|pending
1555|fulton|west|valve|23|shipped
1815|birch|south|valve|18|held
1898|cobalt|north|rotor|85|shipped
1539|cobalt|south|pump|72|held
1586|ionic|east|cable|58|held
2120|birch|north|rotor|67|held
1778|ionic|east|sensor|10|paid
1710|juno|north|panel|80|shipped
1591|birch|east|gasket|89|paid
1663|fulton|east|rotor|28|shipped
1748|cobalt|east|rotor|85|shipped
2047|juno|south|valve|17|paid
1822|fulton|south|pump|71|pending
2021|acme|east|valve|16|paid
1980|harbor|west|cable|41|shipped
1572|harbor|west|valve|27|pending
1493|dorian|north|cable|33|paid
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "juno" AND region = "east" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 49, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "cancel_order", "args": {"order_id": 1453}},
{"tool": "cancel_order", "args": {"order_id": 1459}},
{"tool": "restock", "args": {"item": "panel", "qty": 55}},
{"tool": "restock", "args": {"item": "gasket", "qty": 61}}
]correctagentic.tools.deploy-v1conf 96% · 1.4s · $0.009 · 264 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- notifier: billing
- billing: (none)
- auth-svc: billing
- reports: auth-svc, billing
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "auth-svc" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "billing"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "health_check", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "reports"}}
]correctagentic.tools.ledger-v1conf 99% · 1.7s · $0.012 · 351 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- oscar: $586
- kilo: $823
- delta: $801
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $468 from "kilo" to "delta"
2. pay $177 from "delta" to "oscar"
3. pay $281 from "kilo" to "delta"
4. pay $369 from "oscar" to "delta"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "kilo", "to": "delta", "amount": 468}},
{"tool": "transfer", "args": {"from": "delta", "to": "oscar", "amount": 177}},
{"tool": "transfer", "args": {"from": "kilo", "to": "delta", "amount": 281}},
{"tool": "transfer", "args": {"from": "oscar", "to": "delta", "amount": 369}}
]correctagentic.tools.context-load-v1conf 96% · 1.5s · $0.049 · 894 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (284 records, format: id|customer|region|item|qty|status):
```
2182|gale|east|rotor|89|pending
1451|acme|south|frame|61|shipped
1491|birch|east|pump|76|held
1779|fulton|west|rotor|51|shipped
2138|harbor|east|cable|50|shipped
1533|acme|west|sensor|65|held
2136|fulton|east|valve|63|shipped
2238|ionic|north|valve|17|pending
1733|ember|east|valve|99|pending
1964|harbor|south|gasket|92|paid
2020|dorian|west|valve|96|held
1751|cobalt|north|sensor|84|shipped
1392|acme|east|valve|31|paid
1738|ionic|east|panel|93|shipped
2323|gale|south|rotor|83|held
1670|ionic|south|panel|23|held
2358|harbor|west|sensor|33|shipped
2224|dorian|south|sensor|62|shipped
2013|juno|south|pump|51|shipped
1859|fulton|west|sensor|89|paid
1390|harbor|east|valve|99|paid
2240|cobalt|east|pump|53|pending
1891|fulton|west|gasket|81|pending
2077|fulton|south|panel|89|held
2100|harbor|east|cable|27|pending
2074|ember|north|rotor|36|paid
1852|ember|south|sensor|90|held
1429|cobalt|east|panel|77|shipped
1307|juno|north|pump|39|pending
1946|fulton|west|frame|82|pending
1362|cobalt|west|valve|54|pending
2310|fulton|west|valve|17|paid
1328|juno|north|frame|37|pending
2087|ionic|north|sensor|20|held
1397|dorian|west|gasket|46|shipped
1373|acme|west|frame|49|pending
1382|dorian|east|frame|97|paid
1550|ember|east|sensor|60|held
2049|juno|east|panel|49|held
1934|ember|east|panel|90|held
2381|juno|west|pump|71|pending
2274|fulton|south|frame|99|held
1291|juno|north|rotor|79|pending
1997|birch|west|frame|30|held
1906|acme|north|sensor|37|pending
1591|gale|west|frame|84|shipped
2080|juno|north|valve|11|paid
1446|acme|north|sensor|85|pending
2297|acme|east|frame|58|held
1985|ember|north|pump|27|pending
1410|fulton|south|gasket|62|shipped
1758|ionic|south|pump|72|pending
2321|ember|north|gasket|76|paid
1482|fulton|east|rotor|45|shipped
2336|ember|west|frame|15|held
2328|ember|east|gasket|89|paid
2160|dorian|east|rotor|21|pending
2180|juno|south|gasket|27|pending
2025|ionic|north|frame|96|shipped
1701|dorian|west|rotor|93|pending
1841|cobalt|south|frame|19|held
2193|gale|west|rotor|52|held
1824|cobalt|west|panel|55|shipped
1563|acme|north|rotor|87|held
2153|ionic|west|panel|36|paid
2395|harbor|east|pump|69|paid
2405|juno|east|gasket|51|pending
1484|juno|north|rotor|47|shipped
1646|acme|south|frame|89|held
2214|fulton|east|sensor|98|pending
2299|juno|west|valve|64|shipped
1708|cobalt|south|rotor|77|pending
2024|cobalt|west|pump|50|paid
1358|ember|north|pump|23|paid
1381|gale|south|panel|86|paid
1389|harbor|east|cable|73|paid
2249|gale|west|pump|50|paid
2149|cobalt|north|panel|44|shipped
2368|acme|north|cable|12|shipped
1837|birch|west|gasket|27|paid
1543|fulton|east|gasket|35|shipped
2142|birch|west|frame|53|pending
1961|gale|east|frame|43|shipped
1606|cobalt|west|gasket|12|pending
1457|dorian|west|gasket|68|shipped
2159|gale|west|sensor|59|pending
1940|dorian|north|pump|71|held
1696|gale|south|pump|59|shipped
1960|ember|south|panel|66|held
2262|birch|east|cable|39|shipped
2371|gale|east|frame|99|pending
1464|gale|north|pump|31|held
2204|dorian|north|frame|31|pending
2398|acme|north|frame|89|shipped
2130|gale|west|pump|34|held
1654|harbor|south|rotor|32|held
1715|cobalt|north|panel|99|pending
1615|ember|south|gasket|45|shipped
1545|dorian|east|gasket|71|shipped
1981|juno|north|frame|46|pending
1372|gale|east|pump|98|held
1801|ember|east|panel|91|paid
2338|dorian|north|gasket|78|held
1617|fulton|east|panel|85|pending
2190|fulton|west|pump|92|pending
2112|dorian|west|cable|99|shipped
1897|ionic|east|rotor|89|pending
1455|gale|south|pump|56|pending
1754|ionic|north|frame|60|paid
2021|ember|north|frame|66|pending
1969|ionic|north|valve|31|shipped
2313|cobalt|west|panel|42|shipped
1677|cobalt|north|frame|20|shipped
1297|juno|south|cable|44|pending
1312|juno|east|pump|28|pending
1325|juno|west|cable|55|pending
1874|ionic|west|frame|49|paid
1994|ionic|west|pump|48|pending
1876|acme|west|panel|60|held
2042|gale|south|gasket|26|shipped
2302|ember|north|sensor|31|paid
2370|harbor|east|valve|74|paid
1725|gale|east|sensor|19|paid
1848|gale|west|gasket|76|shipped
1601|harbor|east|sensor|96|paid
1423|fulton|west|valve|78|pending
1377|fulton|south|cable|71|pending
1403|cobalt|south|valve|81|shipped
1684|cobalt|north|gasket|64|shipped
1791|juno|south|panel|49|shipped
1440|fulton|west|valve|13|held
1327|juno|north|sensor|30|held
1458|harbor|east|sensor|79|pending
1991|dorian|east|frame|77|held
2150|cobalt|north|cable|80|paid
1919|birch|west|cable|92|pending
1732|harbor|west|valve|85|pending
1494|harbor|west|frame|10|held
1408|juno|east|frame|67|shipped
2354|acme|south|panel|34|pending
1957|cobalt|north|pump|45|paid
2067|ionic|east|panel|54|pending
1514|ember|west|valve|95|pending
1567|dorian|east|pump|83|pending
1538|gale|south|valve|21|paid
2231|dorian|east|panel|34|held
1691|dorian|north|valve|15|shipped
1516|gale|south|valve|70|shipped
2116|birch|east|valve|61|held
1433|ionic|north|rotor|46|held
1745|acme|north|sensor|93|held
1594|birch|north|frame|17|shipped
1912|juno|south|gasket|49|shipped
1470|ionic|west|valve|32|shipped
1880|ember|east|panel|86|pending
1612|ionic|west|panel|29|paid
2061|gale|south|valve|18|shipped
2047|ionic|west|sensor|66|shipped
1817|dorian|east|rotor|29|paid
1593|fulton|north|cable|58|shipped
1415|ember|east|rotor|36|paid
1952|birch|west|cable|90|shipped
1975|ionic|north|rotor|26|held
1977|acme|north|frame|46|pending
2163|juno|west|panel|71|held
2102|juno|north|frame|90|shipped
1536|juno|south|frame|71|held
2279|harbor|north|sensor|27|shipped
2088|cobalt|north|rotor|63|paid
2268|birch|east|valve|56|held
2243|dorian|east|valve|76|shipped
1805|ember|north|rotor|38|paid
2093|fulton|east|valve|60|shipped
1978|cobalt|west|panel|40|pending
1644|gale|north|valve|56|shipped
2124|birch|north|pump|73|held
1886|harbor|east|cable|75|held
2052|gale|west|gasket|50|held
2023|juno|west|panel|87|paid
2388|acme|south|gasket|90|held
2188|dorian|north|valve|65|paid
1528|juno|south|pump|24|held
1661|acme|east|frame|35|held
1301|juno|north|frame|91|held
2173|juno|south|sensor|51|pending
1588|acme|west|valve|34|pending
1338|juno|north|valve|57|pending
2333|dorian|south|valve|40|paid
1731|fulton|west|cable|59|pending
1872|acme|north|valve|29|pending
2253|harbor|north|gasket|52|pending
1784|fulton|north|frame|96|pending
1913|harbor|west|sensor|19|shipped
2009|birch|east|valve|23|paid
1581|birch|north|cable|27|shipped
2349|harbor|east|rotor|84|pending
1740|ionic|north|cable|77|held
1318|juno|north|valve|60|pending
1468|ember|west|pump|22|paid
1330|juno|south|gasket|74|pending
2271|ember|west|gasket|95|shipped
2032|gale|south|panel|36|paid
1867|ember|south|cable|87|held
1335|juno|north|cable|39|shipped
1955|acme|south|frame|88|pending
2266|birch|north|gasket|58|shipped
2375|birch|north|panel|38|pending
1832|ember|north|pump|78|paid
1353|harbor|east|rotor|55|held
1665|cobalt|north|rotor|60|pending
2210|ionic|west|pump|73|pending
2400|harbor|north|gasket|73|shipped
2256|gale|west|rotor|79|held
2037|juno|west|sensor|65|paid
1522|fulton|north|rotor|92|shipped
2004|juno|west|sensor|58|shipped
1573|cobalt|north|sensor|39|pending
1552|ember|west|cable|19|held
1571|ionic|east|cable|78|paid
1407|harbor|north|gasket|94|held
1574|fulton|west|panel|79|pending
1347|juno|north|pump|72|paid
1498|ionic|north|pump|94|shipped
2281|dorian|west|pump|60|pending
1343|juno|west|sensor|32|pending
2200|birch|east|valve|71|paid
1879|acme|west|pump|49|shipped
2107|juno|west|valve|24|pending
2344|juno|west|frame|65|paid
1860|acme|east|sensor|80|paid
2293|harbor|west|frame|30|paid
2316|gale|north|cable|91|shipped
2096|acme|north|valve|57|paid
1501|birch|east|rotor|19|pending
1902|harbor|south|valve|23|pending
1774|harbor|east|panel|34|shipped
1547|dorian|south|valve|99|pending
1623|cobalt|east|sensor|91|shipped
1555|gale|south|panel|35|pending
2166|ember|west|pump|58|held
1526|gale|east|frame|78|pending
1314|juno|north|rotor|22|paid
1721|ember|west|valve|57|shipped
1595|cobalt|east|frame|29|shipped
1927|cobalt|north|rotor|88|pending
1511|fulton|south|valve|70|shipped
2406|gale|north|valve|39|shipped
2092|cobalt|north|pump|68|shipped
1833|acme|east|rotor|30|shipped
2232|gale|north|panel|54|paid
2056|acme|south|cable|89|held
1386|dorian|south|panel|69|held
2286|dorian|east|rotor|83|shipped
1763|cobalt|east|gasket|22|held
1476|ionic|north|rotor|34|pending
2334|fulton|south|rotor|42|held
1420|cobalt|west|rotor|24|pending
2184|harbor|east|panel|71|shipped
1651|birch|east|panel|31|paid
1767|ember|west|gasket|50|held
1829|dorian|south|cable|33|held
2123|birch|north|valve|88|paid
1512|gale|west|panel|14|pending
2365|ionic|west|gasket|93|held
1562|juno|east|sensor|45|pending
2218|dorian|south|pump|52|paid
1736|acme|south|sensor|31|paid
1962|harbor|east|rotor|70|paid
1369|dorian|east|cable|93|held
2245|harbor|north|frame|20|pending
1766|ionic|west|frame|83|paid
1475|cobalt|west|pump|49|held
1504|juno|north|gasket|68|pending
1922|harbor|east|pump|92|held
2248|harbor|north|cable|14|shipped
1634|fulton|west|gasket|64|shipped
2303|acme|east|sensor|92|shipped
1810|cobalt|east|valve|50|held
1794|fulton|south|valve|16|pending
1808|juno|north|gasket|18|pending
1414|fulton|south|panel|70|shipped
1641|gale|east|sensor|50|pending
1629|ionic|south|rotor|63|pending
1488|ember|east|sensor|42|held
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "juno" AND region = "north" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 69, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "restock", "args": {"item": "rotor", "qty": 79}},
{"tool": "cancel_order", "args": {"order_id": 1307}},
{"tool": "cancel_order", "args": {"order_id": 1318}},
{"tool": "cancel_order", "args": {"order_id": 1328}},
{"tool": "cancel_order", "args": {"order_id": 1338}},
{"tool": "cancel_order", "args": {"order_id": 1504}},
{"tool": "cancel_order", "args": {"order_id": 1808}},
{"tool": "cancel_order", "args": {"order_id": 1981}}
]correctagentic.tools.triage-v1conf 97% · 1.1s · $0.012 · 341 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 8, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → rivera
- data → silva
- auth → haddad
INCIDENTS:
1. "card declined at checkout" (category: payments, priority 4)
2. "export file corrupted" (category: data, priority 6)
3. "export file corrupted" (category: data, priority 6)
4. "refund double-charged" (category: payments, priority 6)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "card declined at checkout", "priority": 4}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "rivera"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "rivera"}}
]correctagentic.tools.deploy-v1conf 90% · 1.1s · $0.010 · 287 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- reports: search
- notifier: (none)
- search: notifier
- auth-svc: search
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "auth-svc" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "health_check", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "reports"}}
]correctagentic.tools.ledger-v1conf 99% · 1.2s · $0.015 · 492 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- kilo: $892
- echo: $701
- lima: $868
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $175 from "echo" to "kilo"
2. pay $557 from "echo" to "lima"
3. pay $324 from "echo" to "lima"
4. pay $101 from "lima" to "echo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "echo", "to": "kilo", "amount": 175}},
{"tool": "top_up_from_reserve", "args": {"account": "echo", "amount": 31}},
{"tool": "transfer", "args": {"from": "echo", "to": "lima", "amount": 557}},
{"tool": "top_up_from_reserve", "args": {"account": "echo", "amount": 324}},
{"tool": "transfer", "args": {"from": "echo", "to": "lima", "amount": 324}},
{"tool": "transfer", "args": {"from": "lima", "to": "echo", "amount": 101}}
]wrongagentic.tools.triage-v1conf 72% · 1.3s · $0.020 · 661 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 7, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- auth → novak
- data → silva
- infra → tanaka
INCIDENTS:
1. "SSO loop on login" (category: auth, priority 9)
2. "export file corrupted" (category: data, priority 5)
3. "SSO loop on login" (category: auth, priority 9)
4. "SSO loop on login" (category: auth, priority 3)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "SSO loop on login", "priority": 9}},
{"tool": "escalate", "args": {"ticket_id": "TCK-1"}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "novak"}},
{"tool": "create_ticket", "args": {"title": "export file corrupted", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "SSO loop on login", "priority": 9}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-1"}},
{"tool": "create_ticket", "args": {"title": "SSO loop on login", "priority": 3}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-4", "resolution": "duplicate of TCK-1"}}
]correctagentic.tools.context-load-v1conf 99% · 2.9s · $0.024 · 440 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (133 records, format: id|customer|region|item|qty|status):
```
1728|cobalt|south|frame|95|shipped
1655|cobalt|south|pump|30|shipped
1374|birch|west|rotor|50|pending
1460|acme|south|sensor|59|pending
1815|acme|south|sensor|36|paid
1587|cobalt|north|pump|21|pending
1781|ionic|south|gasket|72|shipped
1507|juno|west|gasket|50|paid
1512|dorian|east|sensor|39|held
1523|dorian|north|cable|71|held
1569|acme|west|cable|75|pending
1660|ember|south|valve|21|paid
1858|dorian|north|valve|88|pending
1778|juno|west|cable|72|pending
1333|birch|north|valve|86|pending
1422|juno|north|rotor|83|held
1541|ember|west|sensor|30|paid
1328|birch|east|rotor|44|pending
1673|ember|west|pump|72|held
1575|ember|south|cable|56|shipped
1409|gale|west|rotor|68|pending
1779|ember|north|panel|70|pending
1784|juno|east|rotor|69|held
1506|fulton|west|gasket|92|pending
1617|cobalt|south|pump|23|shipped
1745|ionic|north|panel|64|paid
1362|birch|west|rotor|80|pending
1726|gale|north|valve|17|pending
1552|ionic|north|rotor|21|held
1380|acme|west|panel|24|pending
1658|gale|north|cable|75|shipped
1471|cobalt|south|cable|51|shipped
1555|gale|north|sensor|33|shipped
1489|harbor|north|gasket|83|pending
1735|harbor|north|pump|67|paid
1769|cobalt|west|pump|83|pending
1807|dorian|north|gasket|89|pending
1443|dorian|south|valve|29|pending
1439|acme|west|rotor|23|paid
1820|fulton|west|frame|14|held
1787|birch|west|gasket|64|held
1433|juno|south|pump|41|paid
1607|gale|north|pump|52|shipped
1527|gale|north|pump|16|held
1342|birch|east|sensor|57|pending
1663|dorian|west|valve|29|paid
1702|cobalt|west|frame|10|held
1376|birch|east|pump|39|shipped
1551|acme|east|panel|31|pending
1604|cobalt|south|rotor|61|shipped
1862|ionic|south|rotor|16|paid
1426|cobalt|north|panel|91|held
1355|birch|east|pump|70|paid
1851|juno|west|rotor|30|paid
1751|ember|south|frame|52|paid
1628|dorian|south|valve|26|paid
1542|acme|west|cable|64|shipped
1601|ionic|east|frame|68|shipped
1832|dorian|south|valve|24|held
1501|birch|north|pump|15|shipped
1495|ember|north|valve|50|paid
1580|cobalt|north|rotor|83|held
1578|fulton|east|gasket|24|paid
1415|gale|west|rotor|41|held
1366|birch|east|panel|36|paid
1464|fulton|north|valve|58|shipped
1695|harbor|west|pump|44|paid
1830|gale|west|gasket|31|held
1791|ember|east|valve|41|pending
1838|dorian|south|frame|19|pending
1714|acme|east|panel|70|held
1356|birch|east|valve|23|pending
1423|gale|west|frame|87|paid
1776|juno|east|rotor|89|pending
1487|ember|south|pump|19|shipped
1649|acme|south|panel|89|pending
1733|juno|west|pump|15|paid
1447|harbor|west|frame|90|pending
1547|juno|west|cable|57|held
1540|ember|south|frame|80|held
1521|harbor|east|valve|57|shipped
1798|ionic|south|gasket|56|held
1670|ionic|east|rotor|78|shipped
1484|fulton|north|valve|24|pending
1563|ember|west|sensor|20|pending
1643|harbor|east|rotor|71|shipped
1390|gale|east|cable|21|held
1480|fulton|west|frame|44|paid
1378|cobalt|west|cable|30|held
1841|acme|south|panel|48|shipped
1395|ember|north|frame|38|held
1387|acme|west|panel|64|held
1636|harbor|west|rotor|96|held
1707|acme|west|frame|64|shipped
1757|acme|east|pump|64|shipped
1826|cobalt|west|pump|33|held
1453|ember|east|valve|55|held
1819|fulton|north|cable|87|paid
1379|cobalt|south|pump|31|pending
1678|gale|north|gasket|85|shipped
1402|harbor|south|valve|63|paid
1629|dorian|west|gasket|50|shipped
1684|acme|north|frame|39|pending
1340|birch|east|panel|85|held
1801|juno|west|valve|47|held
1720|gale|east|pump|51|held
1795|acme|east|sensor|76|held
1557|dorian|south|gasket|63|pending
1372|birch|east|gasket|24|pending
1401|juno|east|sensor|36|held
1349|birch|west|frame|37|pending
1516|acme|west|rotor|97|pending
1834|ionic|west|pump|56|paid
1763|juno|north|cable|58|held
1623|ember|south|pump|58|paid
1756|acme|west|panel|94|shipped
1690|acme|west|panel|34|pending
1848|harbor|east|cable|71|held
1739|fulton|west|gasket|78|held
1614|acme|west|cable|35|paid
1806|birch|east|rotor|95|shipped
1668|cobalt|south|gasket|37|pending
1490|birch|south|sensor|39|paid
1536|gale|south|frame|32|held
1594|ionic|north|cable|89|paid
1602|cobalt|west|valve|13|shipped
1458|acme|east|cable|84|pending
1534|cobalt|east|frame|60|held
1634|harbor|north|valve|49|paid
1840|acme|west|valve|78|paid
1473|gale|east|frame|11|held
1811|cobalt|west|cable|44|held
1703|juno|west|valve|77|paid
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "birch" AND region = "east" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 47, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "cancel_order", "args": {"order_id": 1328}},
{"tool": "restock", "args": {"item": "sensor", "qty": 57}},
{"tool": "cancel_order", "args": {"order_id": 1356}},
{"tool": "cancel_order", "args": {"order_id": 1372}}
]correctagentic.tools.deploy-v1conf 96% · 1.3s · $0.012 · 388 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- notifier: auth-svc
- auth-svc: search
- gateway: auth-svc, search
- search: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "gateway" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "health_check", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "notifier"}}
]correctagentic.tools.ledger-v1conf 99% · 2.1s · $0.014 · 435 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- echo: $193
- bravo: $650
- kilo: $350
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $297 from "bravo" to "echo"
2. pay $416 from "bravo" to "echo"
3. pay $243 from "echo" to "bravo"
4. pay $595 from "echo" to "kilo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "bravo", "to": "echo", "amount": 297}},
{"tool": "top_up_from_reserve", "args": {"account": "bravo", "amount": 63}},
{"tool": "transfer", "args": {"from": "bravo", "to": "echo", "amount": 416}},
{"tool": "transfer", "args": {"from": "echo", "to": "bravo", "amount": 243}},
{"tool": "transfer", "args": {"from": "echo", "to": "kilo", "amount": 595}}
]correctagentic.tools.context-load-v1conf 98% · 921ms · $0.042 · 833 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (229 records, format: id|customer|region|item|qty|status):
```
1765|ionic|south|rotor|78|paid
1624|cobalt|east|sensor|27|pending
1475|acme|south|valve|31|pending
1299|gale|east|sensor|50|pending
1179|ionic|west|pump|70|held
1342|cobalt|west|gasket|75|held
1108|acme|east|frame|26|shipped
1826|harbor|south|cable|77|shipped
1282|gale|north|gasket|32|paid
1161|dorian|south|cable|70|shipped
1632|fulton|west|sensor|27|pending
1205|fulton|west|sensor|68|paid
1136|cobalt|north|pump|74|pending
1026|juno|east|gasket|21|pending
1544|juno|north|frame|76|pending
1895|ember|north|panel|40|held
1226|acme|north|gasket|84|paid
1650|juno|south|frame|11|shipped
1833|ionic|west|pump|12|pending
1252|gale|south|sensor|70|pending
1487|birch|north|pump|18|paid
1388|juno|west|sensor|74|held
1672|birch|north|gasket|28|paid
1645|harbor|west|sensor|26|shipped
1063|juno|north|sensor|54|pending
1451|acme|west|valve|25|held
1498|ionic|west|rotor|92|held
1492|ember|south|valve|88|held
1236|dorian|south|gasket|24|shipped
1528|juno|south|frame|63|pending
1072|juno|east|valve|95|pending
1460|ember|west|valve|51|held
1340|gale|south|valve|71|paid
1912|ember|north|valve|51|pending
1468|acme|west|cable|76|pending
1400|ember|east|gasket|43|shipped
1638|ionic|east|panel|69|held
1792|cobalt|west|valve|72|paid
1176|harbor|north|sensor|27|held
1683|fulton|north|panel|43|held
1358|gale|east|sensor|15|shipped
1314|birch|south|valve|17|pending
1647|birch|east|panel|83|paid
1159|cobalt|east|sensor|35|pending
1702|cobalt|south|frame|77|pending
1964|gale|west|valve|92|held
1665|birch|east|pump|14|shipped
1044|juno|west|gasket|60|pending
1924|dorian|west|valve|73|held
1723|fulton|south|pump|41|pending
1145|gale|north|sensor|12|shipped
1948|fulton|east|panel|56|paid
1374|dorian|north|cable|92|shipped
1481|ionic|east|valve|96|shipped
1129|cobalt|east|cable|60|pending
1050|juno|east|panel|70|paid
1865|gale|south|valve|99|paid
1806|gale|east|gasket|16|held
1689|cobalt|east|pump|89|shipped
1893|harbor|west|pump|47|shipped
1879|birch|west|gasket|44|pending
1769|acme|north|rotor|92|held
1086|ionic|north|sensor|99|pending
1589|juno|north|valve|61|pending
1326|harbor|south|valve|36|shipped
1799|birch|east|cable|56|held
1469|juno|east|pump|17|paid
1600|gale|south|frame|74|paid
1930|acme|north|sensor|81|pending
1852|ember|east|panel|13|pending
1112|acme|east|cable|55|held
1306|ionic|east|pump|93|pending
1150|harbor|west|valve|70|held
1794|juno|east|rotor|37|pending
1126|acme|west|pump|37|pending
1394|ionic|south|cable|42|paid
1558|birch|south|rotor|51|shipped
1320|birch|east|gasket|51|paid
1741|ember|north|gasket|31|shipped
1829|gale|north|pump|89|shipped
1253|cobalt|west|sensor|75|pending
1539|birch|west|panel|24|paid
1221|ember|east|valve|56|shipped
1025|juno|east|cable|78|held
1407|ionic|south|rotor|85|paid
1230|ember|north|gasket|98|shipped
1117|acme|south|valve|67|held
1676|dorian|west|pump|82|paid
1134|fulton|south|sensor|77|paid
1548|ember|north|panel|10|paid
1613|ionic|east|gasket|15|held
1085|acme|north|pump|42|held
1416|birch|south|cable|32|pending
1906|dorian|south|cable|83|paid
1074|juno|west|valve|91|pending
1905|cobalt|south|cable|39|held
1885|acme|west|rotor|35|pending
1422|dorian|west|rotor|96|pending
1091|acme|north|cable|87|held
1727|birch|north|cable|67|shipped
1452|fulton|north|cable|14|paid
1630|ionic|south|valve|89|shipped
1887|ionic|north|valve|93|held
1956|dorian|south|panel|19|pending
1102|juno|north|cable|65|shipped
1572|birch|east|gasket|83|shipped
1562|ember|west|gasket|95|held
1607|harbor|west|cable|50|held
1098|cobalt|east|sensor|65|paid
1555|ember|east|cable|60|pending
1873|ionic|south|sensor|30|paid
1659|juno|south|cable|40|pending
1705|acme|west|valve|73|pending
1437|cobalt|south|frame|62|pending
1376|birch|east|frame|82|held
1795|birch|north|frame|70|shipped
1266|ionic|south|panel|58|pending
1810|cobalt|south|gasket|28|held
1518|juno|north|gasket|35|held
1778|cobalt|south|sensor|25|shipped
1774|fulton|west|gasket|70|paid
1315|acme|east|rotor|44|pending
1169|harbor|west|gasket|85|held
1225|acme|west|valve|83|shipped
1289|acme|south|pump|44|shipped
1212|dorian|south|valve|80|paid
1890|dorian|north|sensor|41|shipped
1853|acme|west|sensor|52|held
1158|dorian|west|pump|30|held
1384|cobalt|west|valve|35|pending
1525|acme|east|gasket|74|pending
1919|cobalt|south|valve|15|shipped
1784|harbor|west|pump|28|held
1351|fulton|west|frame|92|held
1429|ionic|south|sensor|24|paid
1019|juno|east|panel|19|pending
1347|cobalt|north|gasket|31|pending
1463|ionic|north|cable|13|held
1272|juno|north|pump|73|pending
1840|juno|south|pump|44|pending
1191|juno|north|rotor|46|shipped
1760|ionic|south|rotor|25|held
1413|ember|east|panel|43|paid
1121|cobalt|south|panel|99|pending
1817|ionic|east|panel|27|held
1444|ionic|south|rotor|41|held
1575|dorian|west|sensor|13|pending
1869|ionic|south|valve|24|paid
1209|ember|north|frame|36|pending
1814|dorian|west|gasket|76|shipped
1308|acme|south|gasket|20|held
1584|dorian|south|panel|58|paid
1673|harbor|west|valve|18|shipped
1663|gale|south|frame|42|held
1257|ember|south|cable|61|paid
1618|fulton|west|panel|32|pending
1870|gale|west|frame|66|pending
1914|fulton|north|cable|34|shipped
1219|ember|east|frame|67|pending
1022|juno|west|cable|80|pending
1847|harbor|south|valve|92|shipped
1080|ember|south|cable|19|paid
1953|ember|east|gasket|54|paid
1260|juno|south|panel|74|pending
1513|gale|south|valve|38|paid
1763|harbor|east|rotor|65|shipped
1037|juno|east|rotor|17|pending
1378|acme|west|pump|36|pending
1858|birch|east|frame|87|pending
1511|birch|south|panel|75|paid
1505|acme|south|pump|81|held
1138|dorian|north|rotor|81|paid
1031|juno|east|gasket|20|shipped
1936|gale|south|sensor|48|pending
1696|ember|west|rotor|10|shipped
1057|juno|east|valve|87|pending
1700|harbor|south|cable|38|shipped
1333|ember|north|valve|59|shipped
1748|dorian|east|pump|45|pending
1244|dorian|south|panel|64|held
1275|birch|east|valve|98|paid
1717|dorian|west|rotor|85|paid
1453|birch|south|valve|95|held
1734|fulton|east|rotor|38|held
1405|ember|south|rotor|96|held
1317|harbor|west|frame|63|shipped
1200|fulton|south|gasket|83|shipped
1028|juno|north|sensor|67|pending
1739|juno|north|panel|33|paid
1694|gale|south|cable|75|paid
1311|gale|west|frame|48|shipped
1941|juno|east|rotor|62|paid
1534|fulton|east|frame|98|paid
1294|cobalt|north|cable|48|shipped
1789|juno|west|cable|41|paid
1643|birch|south|gasket|31|pending
1901|juno|south|valve|82|paid
1594|dorian|south|panel|10|held
1959|fulton|east|gasket|42|pending
1067|juno|east|valve|49|held
1612|harbor|south|valve|37|paid
1958|birch|east|gasket|74|paid
1565|ionic|east|frame|45|paid
1077|juno|east|frame|46|paid
1238|acme|west|frame|38|held
1675|juno|south|sensor|33|held
1712|juno|south|pump|89|pending
1653|ember|west|panel|68|shipped
1361|ember|west|cable|96|paid
1813|gale|east|sensor|81|pending
1245|birch|west|valve|39|shipped
1325|harbor|south|frame|26|shipped
1432|ember|west|cable|67|shipped
1180|acme|west|pump|90|paid
1184|cobalt|north|gasket|92|pending
1271|ionic|east|cable|18|shipped
1164|birch|west|valve|88|pending
1970|juno|east|panel|19|pending
1152|cobalt|north|gasket|40|held
1197|cobalt|south|gasket|85|pending
1755|acme|east|valve|63|held
1971|fulton|north|cable|22|held
1350|gale|south|rotor|81|paid
1276|gale|west|panel|99|paid
1536|acme|east|valve|94|paid
1582|birch|east|sensor|21|held
1819|birch|north|panel|78|pending
1124|cobalt|south|cable|88|held
1367|ember|east|panel|87|pending
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "juno" AND region = "east" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 49, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "cancel_order", "args": {"order_id": 1019}},
{"tool": "cancel_order", "args": {"order_id": 1026}},
{"tool": "cancel_order", "args": {"order_id": 1037}},
{"tool": "restock", "args": {"item": "valve", "qty": 87}},
{"tool": "restock", "args": {"item": "valve", "qty": 95}},
{"tool": "cancel_order", "args": {"order_id": 1794}},
{"tool": "cancel_order", "args": {"order_id": 1970}}
]correctagentic.tools.triage-v1conf 96% · 989ms · $0.010 · 262 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 9, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- infra → rivera
- data → dubois
- payments → okafor
INCIDENTS:
1. "API latency spikes" (category: infra, priority 7)
2. "records missing after import" (category: data, priority 8)
3. "records missing after import" (category: data, priority 8)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 7}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "rivera"}},
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 8}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 8}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}}
]correctagentic.tools.deploy-v1conf 92% · 1.1s · $0.008 · 225 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- gateway: billing, search
- billing: notifier, search
- search: notifier
- notifier: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "billing" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "billing"}},
{"tool": "health_check", "args": {"service": "billing"}},
{"tool": "deploy", "args": {"service": "gateway"}}
]correctagentic.tools.ledger-v1conf 99% · 1.1s · $0.012 · 348 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- tango: $559
- delta: $723
- kilo: $590
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $198 from "tango" to "kilo"
2. pay $447 from "delta" to "tango"
3. pay $173 from "kilo" to "tango"
4. pay $222 from "kilo" to "tango"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "tango", "to": "kilo", "amount": 198}},
{"tool": "transfer", "args": {"from": "delta", "to": "tango", "amount": 447}},
{"tool": "transfer", "args": {"from": "kilo", "to": "tango", "amount": 173}},
{"tool": "transfer", "args": {"from": "kilo", "to": "tango", "amount": 222}}
]correctagentic.tools.context-load-v1conf 96% · 1.1s · $0.045 · 784 tok
question
You are an order-operations agent working strictly through tool calls.
TOOL CATALOG:
- restock(item: string, qty: int) — reorders stock for a large pending order
- cancel_order(order_id: int) — cancels a small pending order
- ship_order(order_id: int) — ships a paid order (out of scope here)
- refund(order_id: int, amount: int) — refunds a customer (out of scope here)
- notify_customer(customer: string, message: string) — sends a notification (not required by this policy)
ORDER LEDGER (272 records, format: id|customer|region|item|qty|status):
```
2331|fulton|south|frame|67|shipped
2032|birch|south|frame|31|pending
2313|harbor|south|rotor|68|held
1787|gale|south|cable|85|pending
2426|harbor|north|cable|94|paid
1718|ember|north|pump|97|pending
2299|cobalt|east|pump|10|held
1982|acme|west|panel|71|shipped
1821|juno|south|frame|39|shipped
2226|harbor|north|cable|11|paid
1416|fulton|south|rotor|79|pending
2269|ionic|west|frame|29|held
1673|cobalt|east|pump|27|held
2043|gale|south|gasket|88|shipped
1477|birch|east|rotor|20|paid
2446|fulton|east|sensor|41|paid
2456|juno|south|sensor|11|pending
2202|cobalt|south|rotor|71|paid
1703|juno|west|pump|66|paid
1446|fulton|south|panel|10|pending
1497|fulton|west|pump|19|pending
1622|cobalt|south|valve|58|shipped
1866|juno|west|frame|84|held
2351|fulton|east|valve|47|held
1876|gale|west|panel|88|shipped
2102|harbor|south|panel|25|shipped
1728|fulton|south|gasket|61|held
1888|acme|south|gasket|73|shipped
2130|harbor|south|valve|79|paid
1917|juno|west|panel|43|pending
1805|fulton|west|rotor|70|held
2311|ionic|east|sensor|90|paid
2490|acme|east|gasket|37|held
1795|birch|south|sensor|85|pending
1850|ember|south|rotor|47|held
2257|acme|east|gasket|80|held
2152|acme|west|panel|31|paid
1665|ionic|north|valve|92|shipped
1933|juno|west|panel|24|shipped
2022|fulton|east|panel|69|shipped
2001|dorian|north|valve|97|paid
2450|fulton|west|sensor|29|pending
2462|juno|west|valve|33|held
2340|cobalt|south|valve|21|held
1738|birch|north|valve|81|paid
1996|ember|west|panel|22|held
2394|ionic|west|frame|65|pending
1781|birch|east|sensor|69|pending
2183|juno|north|sensor|35|pending
2297|ember|south|pump|35|paid
2061|juno|south|frame|15|shipped
2019|ember|north|valve|83|pending
2290|ionic|south|pump|78|held
2031|ember|east|gasket|31|shipped
1422|fulton|west|pump|70|pending
1490|juno|north|sensor|78|held
2090|fulton|south|gasket|78|paid
2177|harbor|south|gasket|94|shipped
1838|acme|south|gasket|35|pending
1716|cobalt|north|frame|98|paid
1652|ember|west|pump|76|held
2454|acme|east|rotor|59|shipped
1549|birch|south|panel|13|paid
2286|fulton|south|sensor|75|paid
2281|fulton|west|valve|70|pending
2068|acme|south|panel|68|shipped
1409|fulton|south|frame|34|paid
1594|dorian|west|panel|81|paid
2370|gale|east|valve|88|shipped
2046|dorian|south|panel|53|pending
1631|fulton|west|panel|59|pending
1564|ember|south|frame|47|paid
1967|cobalt|east|cable|97|pending
1847|dorian|west|rotor|52|paid
2487|acme|east|pump|39|shipped
2319|dorian|north|valve|99|pending
1438|fulton|north|pump|14|pending
2439|cobalt|west|panel|98|held
1706|juno|south|rotor|23|shipped
1555|birch|south|panel|42|paid
1822|dorian|west|valve|47|paid
2347|juno|east|frame|78|held
1636|acme|west|frame|18|pending
1515|birch|north|valve|23|paid
2047|birch|south|rotor|26|pending
1431|fulton|south|valve|68|pending
1599|cobalt|west|frame|12|held
2459|cobalt|south|frame|59|held
2378|acme|south|gasket|97|pending
1775|dorian|south|cable|79|held
2197|gale|north|sensor|88|pending
2243|ionic|east|cable|33|shipped
2232|dorian|east|panel|86|shipped
1939|acme|east|cable|11|held
1815|acme|south|frame|35|paid
2097|birch|west|gasket|17|paid
1616|ember|north|frame|47|pending
1502|gale|north|panel|27|shipped
1546|juno|north|panel|19|shipped
2066|acme|south|sensor|19|shipped
2017|ember|north|pump|61|held
1439|fulton|south|rotor|79|paid
1883|gale|north|rotor|72|pending
1910|harbor|east|panel|95|shipped
1474|birch|east|panel|38|shipped
2146|juno|west|valve|20|pending
1536|juno|south|rotor|18|paid
2133|gale|north|sensor|72|held
2149|harbor|south|pump|72|shipped
1604|harbor|west|pump|88|pending
1533|ember|north|pump|81|paid
2261|birch|north|rotor|74|pending
2054|ember|east|frame|78|paid
1801|dorian|west|sensor|16|pending
1904|harbor|south|cable|33|held
2029|acme|south|valve|38|shipped
2139|birch|east|frame|90|pending
2389|harbor|south|pump|23|paid
1610|gale|north|sensor|58|paid
2381|ember|south|rotor|96|held
2002|gale|south|valve|89|shipped
1727|cobalt|north|frame|75|shipped
2158|fulton|south|rotor|88|shipped
1486|fulton|north|cable|71|paid
2210|gale|south|rotor|63|pending
1895|harbor|south|sensor|64|shipped
1529|cobalt|east|valve|47|held
2383|birch|south|valve|40|shipped
2393|ionic|west|sensor|72|shipped
1661|birch|west|frame|68|pending
1570|ember|west|frame|88|held
2425|birch|south|frame|83|held
1465|fulton|north|frame|59|pending
2452|acme|south|rotor|98|paid
1765|cobalt|west|gasket|26|pending
1769|cobalt|south|cable|78|paid
1818|fulton|north|pump|34|held
1573|birch|west|sensor|41|pending
2081|ember|west|cable|22|held
1840|fulton|south|sensor|65|paid
1853|ionic|east|frame|28|pending
1483|gale|west|valve|61|shipped
1489|juno|east|gasket|79|held
1829|dorian|north|sensor|30|pending
1945|juno|north|frame|50|shipped
1749|gale|north|pump|19|pending
1979|birch|north|pump|48|paid
1771|ember|east|gasket|14|held
1425|fulton|south|cable|40|shipped
1778|harbor|south|pump|90|pending
1664|ember|west|cable|14|held
1942|fulton|east|gasket|92|pending
2384|ember|east|cable|98|shipped
2500|ionic|south|panel|27|paid
2246|gale|south|pump|45|held
1403|fulton|west|panel|20|pending
1742|juno|north|pump|40|paid
2049|acme|south|cable|75|paid
2477|gale|east|cable|19|held
2433|harbor|east|pump|43|paid
1950|gale|south|valve|76|held
1522|acme|west|pump|92|held
2324|birch|north|valve|79|held
1626|dorian|north|sensor|75|held
2264|ionic|east|pump|11|paid
1915|harbor|west|valve|72|paid
2004|ember|west|gasket|97|shipped
2184|acme|west|sensor|48|paid
1788|fulton|north|valve|16|held
2074|dorian|east|panel|60|paid
2238|gale|north|cable|10|paid
2164|gale|north|panel|51|shipped
2336|gale|north|sensor|58|pending
1542|acme|north|rotor|12|paid
1587|fulton|south|sensor|44|held
2190|birch|north|sensor|14|pending
1558|ionic|south|rotor|35|paid
2308|gale|east|rotor|85|shipped
2150|fulton|east|sensor|16|pending
1735|juno|east|sensor|19|held
1984|juno|south|valve|97|pending
1758|gale|north|panel|30|shipped
1992|gale|south|pump|80|paid
1683|fulton|west|rotor|21|pending
1833|birch|west|valve|74|paid
1647|gale|north|gasket|98|held
2409|cobalt|south|frame|77|pending
1860|birch|south|cable|56|shipped
2036|cobalt|south|sensor|88|paid
2115|ionic|west|panel|29|shipped
2253|harbor|east|pump|68|shipped
2006|acme|west|gasket|50|pending
1578|ionic|east|pump|73|held
1995|gale|west|pump|39|pending
1470|fulton|south|gasket|47|held
2255|ember|east|valve|50|shipped
2271|acme|south|panel|54|paid
2395|ionic|north|valve|17|held
1450|fulton|west|panel|24|pending
2157|gale|west|pump|96|shipped
1696|gale|east|rotor|22|pending
1671|gale|west|gasket|70|held
2472|harbor|east|valve|34|shipped
1755|fulton|north|panel|62|shipped
2367|juno|south|panel|25|paid
2220|ember|west|rotor|43|shipped
2134|fulton|west|pump|14|pending
1726|fulton|west|pump|86|paid
2356|ionic|west|frame|19|pending
2495|fulton|west|sensor|97|paid
1709|ember|north|gasket|26|paid
1561|cobalt|north|gasket|74|pending
2171|ionic|east|valve|11|pending
1583|ember|east|panel|19|paid
1551|ionic|east|valve|95|held
1957|ionic|south|pump|97|paid
2213|gale|south|gasket|37|held
1589|harbor|north|frame|96|held
2239|juno|north|pump|78|held
1460|fulton|south|sensor|53|pending
1399|fulton|south|valve|41|pending
1590|acme|east|valve|97|held
1690|fulton|west|pump|93|shipped
2298|acme|north|gasket|80|shipped
2482|fulton|east|valve|21|shipped
2035|gale|east|sensor|33|paid
2122|birch|north|valve|45|pending
2415|acme|south|sensor|75|paid
2343|harbor|south|cable|10|pending
1900|ionic|west|pump|59|paid
1971|gale|west|sensor|70|shipped
1930|juno|south|gasket|35|shipped
1810|ember|north|cable|30|pending
1641|ember|east|sensor|74|paid
1972|ionic|south|pump|46|shipped
1869|birch|west|sensor|55|held
1794|ember|east|gasket|43|pending
1457|fulton|south|sensor|34|shipped
1994|gale|west|panel|15|paid
2315|acme|east|sensor|53|held
2278|ember|north|cable|39|shipped
2304|gale|west|gasket|75|paid
2109|birch|east|gasket|12|held
1991|birch|east|rotor|70|held
1510|ember|south|pump|19|pending
1520|acme|south|rotor|36|paid
2276|ionic|south|cable|25|held
2402|dorian|north|rotor|94|shipped
2467|gale|west|sensor|99|shipped
1511|ionic|west|gasket|37|pending
1960|acme|west|frame|81|shipped
1492|harbor|north|panel|50|held
2479|harbor|east|panel|79|shipped
2085|cobalt|east|sensor|71|paid
2260|ember|north|valve|46|paid
1722|juno|east|cable|41|shipped
2011|harbor|west|valve|32|held
2208|gale|north|cable|34|pending
1802|harbor|west|cable|88|pending
2419|fulton|south|valve|41|shipped
1924|fulton|west|pump|15|shipped
2363|dorian|west|sensor|28|shipped
2376|acme|south|panel|93|pending
1657|acme|west|cable|91|pending
1505|gale|east|rotor|58|shipped
1931|acme|east|cable|57|shipped
2399|fulton|south|valve|18|held
1935|cobalt|south|cable|71|shipped
1830|birch|east|valve|70|pending
1677|ember|south|valve|17|held
1952|ember|east|rotor|83|held
2124|ember|west|panel|36|paid
```
POLICY (apply exactly):
- Consider ONLY orders where customer = "fulton" AND region = "south" AND status = "pending".
- Process those orders in ASCENDING order id.
- For each: if qty > 55, call restock(item, qty) with that order's item and qty; otherwise call cancel_order(order_id).
- Call nothing else. Every other record is irrelevant no matter how similar it looks.
Output the full ordered sequence of tool calls as a JSON array of {"tool": string, "args": object}.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "cancel_order", "args": {"order_id": 1399}},
{"tool": "restock", "args": {"item": "rotor", "qty": 79}},
{"tool": "restock", "args": {"item": "valve", "qty": 68}},
{"tool": "cancel_order", "args": {"order_id": 1446}},
{"tool": "cancel_order", "args": {"order_id": 1460}}
]correctagentic.tools.triage-v1conf 96% · 1.4s · $0.013 · 354 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 9, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- infra → tanaka
- auth → haddad
- data → dubois
INCIDENTS:
1. "webhooks not delivered" (category: infra, priority 2)
2. "cannot reset password" (category: auth, priority 6)
3. "dashboard shows stale numbers" (category: data, priority 5)
4. "dashboard shows stale numbers" (category: data, priority 5)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "webhooks not delivered", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "tanaka"}},
{"tool": "create_ticket", "args": {"title": "cannot reset password", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "haddad"}},
{"tool": "create_ticket", "args": {"title": "dashboard shows stale numbers", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "dubois"}},
{"tool": "create_ticket", "args": {"title": "dashboard shows stale numbers", "priority": 5}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-4", "resolution": "duplicate of TCK-3"}}
]correctagentic.tools.ledger-v1conf 99% · 1.9s · $0.013 · 392 tok
question
You are a treasury agent operating strictly through tool calls.
TOOL CATALOG:
- transfer(from: string, to: string, amount: int) — moves funds between accounts; FAILS if it would overdraw the source
- top_up_from_reserve(account: string, amount: int) — adds funds to an account from the corporate reserve
- get_balance(account: string) — reads a balance (you already have all balances below — reads are unnecessary and forbidden)
- freeze_account(account: string) — compliance freeze (not part of this task)
- convert_currency(account: string, currency: string) — FX conversion (all amounts are already in USD)
OPENING BALANCES:
- bravo: $227
- delta: $763
- kilo: $617
PAYMENT INSTRUCTIONS (execute in exactly this order):
1. pay $475 from "delta" to "kilo"
2. pay $531 from "kilo" to "delta"
3. pay $354 from "kilo" to "delta"
4. pay $181 from "bravo" to "kilo"
RULES:
- transfer() fails on overdraft. If a payment would overdraw its source account at the moment of execution, first call top_up_from_reserve() on the source with EXACTLY the shortfall (no more, no less), then execute the transfer.
- Track balances as they change: earlier payments affect later ones.
- Never call tools that are not needed.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "transfer", "args": {"from": "delta", "to": "kilo", "amount": 475}},
{"tool": "transfer", "args": {"from": "kilo", "to": "delta", "amount": 531}},
{"tool": "transfer", "args": {"from": "kilo", "to": "delta", "amount": 354}},
{"tool": "transfer", "args": {"from": "bravo", "to": "kilo", "amount": 181}}
]correctagentic.tools.deploy-v1conf 88% · 920ms · $0.008 · 223 tok
question
You are a release agent operating strictly through tool calls.
TOOL CATALOG:
- deploy(service: string) — deploys a service; FAILS if any dependency is not yet deployed
- health_check(service: string) — runs a post-deploy health probe
- rollback(service: string) — reverts a bad deploy (nothing fails in this scenario)
- scale(service: string, replicas: int) — changes replica count (out of scope)
- restart(service: string) — restarts a service (out of scope)
SERVICES AND THEIR DEPENDENCIES (a service can only be deployed after ALL its dependencies):
- billing: gateway
- gateway: search
- notifier: gateway
- search: (none)
POLICY:
- Deploy in waves: in each wave, deploy every service whose dependencies are already deployed, in alphabetical order; repeat until all services are deployed.
- The service "notifier" is flagged unstable: call health_check on it immediately after deploying it.
- Call nothing else.
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "deploy", "args": {"service": "search"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "billing"}},
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "health_check", "args": {"service": "notifier"}}
]correctagentic.tools.triage-v1conf 82% · 1.0s · $0.021 · 686 tok
question
You operate a support desk strictly through tool calls.
TOOL CATALOG:
- create_ticket(title: string, priority: int) — opens a ticket; IDs are assigned sequentially: the 1st created ticket is "TCK-1", the 2nd "TCK-2", etc.
- assign(ticket_id: string, agent: string) — assigns an open ticket to an agent
- escalate(ticket_id: string) — marks a ticket as escalated
- close_ticket(ticket_id: string, resolution: string) — closes a ticket with a resolution note
- send_email(to: string, body: string) — sends an email (not part of the triage policy)
- archive_ticket(ticket_id: string) — archives a closed ticket (nightly job does this automatically)
- set_reminder(ticket_id: string, hours: int) — sets a follow-up reminder
ROUTING POLICY (apply exactly, in this order, for each incident, processing incidents in the order listed):
1. Create a ticket for the incident (title = the incident text verbatim, priority as given).
2. If the incident is an exact duplicate of an earlier incident in this list, close its ticket immediately with resolution "duplicate of <ID of the earlier ticket>" and do nothing else for it.
3. Otherwise, if priority ≥ 7, escalate the ticket BEFORE assigning it.
4. Assign the ticket to the agent responsible for its category.
CATEGORY → AGENT:
- payments → silva
- infra → chen
- auth → tanaka
INCIDENTS:
1. "invoice total wrong" (category: payments, priority 5)
2. "API latency spikes" (category: infra, priority 6)
3. "API latency spikes" (category: infra, priority 6)
4. "invoice total wrong" (category: payments, priority 7)
Output the full ordered sequence of tool calls needed to accomplish the goal, as a JSON array of objects {"tool": string, "args": object}. Use exactly the tool and argument names from the catalog. Do not call any tool that is not required.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
[
{"tool": "create_ticket", "args": {"title": "invoice total wrong", "priority": 5}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "silva"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 6}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "chen"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 6}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-3", "resolution": "duplicate of TCK-2"}},
{"tool": "create_ticket", "args": {"title": "invoice total wrong", "priority": 7}},
{"tool": "escalate", "args": {"ticket_id": "TCK-4"}},
{"tool": "assign", "args": {"ticket_id": "TCK-4", "agent": "silva"}}
]correctagentic.tools.context-load-v1anchorconf 98% · 2.2s · $0.038 · 767 tok
model answer:
[
{"tool": "restock", "args": {"item": "gasket", "qty": 60}},
{"tool": "cancel_order", "args": {"order_id": 1368}},
{"tool": "cancel_order", "args": {"order_id": 1384}},
{"tool": "restock", "args": {"item": "frame", "qty": 58}},
{"tool": "restock", "args": {"item": "rotor", "qty": 75}},
{"tool": "cancel_order", "args": {"order_id": 1682}}
]correctagentic.tools.deploy-v1anchorconf 90% · 1.2s · $0.009 · 242 tok
model answer:
[
{"tool": "deploy", "args": {"service": "notifier"}},
{"tool": "deploy", "args": {"service": "auth-svc"}},
{"tool": "deploy", "args": {"service": "gateway"}},
{"tool": "health_check", "args": {"service": "gateway"}},
{"tool": "deploy", "args": {"service": "reports"}}
]correctagentic.tools.ledger-v1anchorconf 99% · 1.3s · $0.018 · 609 tok
model answer:
[
{"tool": "top_up_from_reserve", "args": {"account": "delta", "amount": 134}},
{"tool": "transfer", "args": {"from": "delta", "to": "bravo", "amount": 451}},
{"tool": "transfer", "args": {"from": "bravo", "to": "delta", "amount": 292}},
{"tool": "transfer", "args": {"from": "delta", "to": "tango", "amount": 97}},
{"tool": "top_up_from_reserve", "args": {"account": "delta", "amount": 345}},
{"tool": "transfer", "args": {"from": "delta", "to": "bravo", "amount": 540}},
{"tool": "transfer", "args": {"from": "tango", "to": "bravo", "amount": 125}}
]correctagentic.tools.triage-v1anchorconf 96% · 3.3s · $0.013 · 372 tok
model answer:
[
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 9}},
{"tool": "escalate", "args": {"ticket_id": "TCK-1"}},
{"tool": "assign", "args": {"ticket_id": "TCK-1", "agent": "tanaka"}},
{"tool": "create_ticket", "args": {"title": "records missing after import", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-2", "agent": "novak"}},
{"tool": "create_ticket", "args": {"title": "refund double-charged", "priority": 2}},
{"tool": "assign", "args": {"ticket_id": "TCK-3", "agent": "okafor"}},
{"tool": "create_ticket", "args": {"title": "API latency spikes", "priority": 9}},
{"tool": "close_ticket", "args": {"ticket_id": "TCK-4", "resolution": "duplicate of TCK-1"}}
]code 30/30 correct
correctcode.trace.nested-v1conf 99% · 1.2s · $0.024 · 918 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 8):
for j in range(1, 6):
if j == 4 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 3 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
204correctcode.trace.js-v1conf 100% · 1.6s · $0.007 · 238 tok
question
What does this JavaScript program log? ```js const arr = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]; const out = arr .map(n => n * 6) .filter(n => n % 5 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
90correctcode.trace.nested-v1conf 99% · 3.2s · $0.016 · 596 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 6):
for j in range(1, 5):
if j == 3 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 4 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
116correctcode.trace.python-v1conf 100% · 1.1s · $0.008 · 302 tok
question
What does this Python program print?
```python
total = 0
v = 7
while total + v <= 35:
if v % 3 != 0:
total += v
v += 9
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
23correctcode.trace.js-v1conf 100% · 1.1s · $0.006 · 212 tok
question
What does this JavaScript program log? ```js const arr = [8, 9, 10, 11, 12, 13, 14, 15, 16, 17]; const out = arr .map(n => n * 3) .filter(n => n % 5 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
75correctcode.trace.python-v1conf 99% · 1.1s · $0.009 · 319 tok
question
Execute this Python snippet mentally. What is printed?
```python
total = 0
v = 10
while total + v <= 103:
if v % 6 != 0:
total += v
v += 8
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
70correctcode.trace.nested-v1conf 99% · 1.1s · $0.023 · 890 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 8):
for j in range(1, 8):
if j == 3 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 4 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
291correctcode.trace.js-v1conf 100% · 1.4s · $0.007 · 250 tok
question
What does this JavaScript program log? ```js const arr = [7, 8, 9, 10, 11, 12]; const out = arr .map(n => n * 2) .filter(n => n % 4 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
60correctcode.trace.python-v1conf 100% · 1.9s · $0.008 · 277 tok
question
Trace the following Python code and give its exact output.
```python
total = 0
v = 5
while total + v <= 36:
if v % 6 != 0:
total += v
v += 8
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
18correctcode.trace.nested-v1conf 99% · 1.5s · $0.025 · 946 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 7):
for j in range(1, 8):
if j == 3 and i % 2 == 0:
break
if (i + j) % 4 == 0:
continue
total += i * 2 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
185correctcode.trace.js-v1conf 99% · 2.8s · $0.006 · 190 tok
question
What does this JavaScript program log? ```js const arr = [5, 6, 7, 8, 9, 10, 11]; const out = arr .map(n => n * 3) .filter(n => n % 4 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
24correctcode.trace.python-v1conf 99% · 1.1s · $0.012 · 446 tok
question
Trace the following Python code and give its exact output.
```python
total = 0
v = 3
while total + v <= 80:
if v % 7 != 0:
total += v
v += 7
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
54correctcode.trace.js-v1conf 100% · 1.5s · $0.011 · 413 tok
question
Evaluate the following JavaScript. What number is logged to the console? ```js const arr = [7, 8, 9, 10, 11, 12, 13, 14, 15]; const out = arr .map(n => n * 6) .filter(n => n % 4 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
264correctcode.trace.nested-v1conf 99% · 1.3s · $0.016 · 596 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 5):
for j in range(1, 6):
if j == 4 and i % 2 == 0:
break
if (i + j) % 3 == 0:
continue
total += i * 3 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
110correctcode.trace.python-v1conf 99% · 1.6s · $0.013 · 489 tok
question
What does this Python program print?
```python
total = 0
v = 9
while total + v <= 84:
if v % 6 != 0:
total += v
v += 3
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
72correctcode.trace.nested-v1conf 99% · 837ms · $0.024 · 922 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 7):
for j in range(1, 8):
if j == 6 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 4 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
315correctcode.trace.js-v1conf 100% · 1.2s · $0.010 · 349 tok
question
Evaluate the following JavaScript. What number is logged to the console? ```js const arr = [2, 3, 4, 5, 6, 7, 8]; const out = arr .map(n => n * 6) .filter(n => n % 4 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
120correctcode.trace.python-v1conf 99% · 1.1s · $0.008 · 287 tok
question
What does this Python program print?
```python
total = 0
v = 1
while total + v <= 116:
if v % 5 != 0:
total += v
v += 7
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
96correctcode.trace.nested-v1conf 99% · 1.3s · $0.020 · 761 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 7):
for j in range(1, 6):
if j == 4 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 2 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
114correctcode.trace.js-v1conf 100% · 1.1s · $0.009 · 320 tok
question
What does this JavaScript program log? ```js const arr = [4, 5, 6, 7, 8, 9, 10, 11, 12]; const out = arr .map(n => n * 2) .filter(n => n % 4 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
80correctcode.trace.python-v1conf 99% · 1.2s · $0.008 · 308 tok
question
Execute this Python snippet mentally. What is printed?
```python
total = 0
v = 5
while total + v <= 34:
if v % 5 != 0:
total += v
v += 6
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
28correctcode.trace.nested-v1conf 99% · 1.4s · $0.023 · 880 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 8):
for j in range(1, 7):
if j == 3 and i % 2 == 0:
break
if (i + j) % 2 == 0:
continue
total += i * 5 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
351correctcode.trace.js-v1conf 99% · 1.0s · $0.006 · 182 tok
question
What does this JavaScript program log? ```js const arr = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]; const out = arr .map(n => n * 2) .filter(n => n % 2 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
228correctcode.trace.python-v1conf 99% · 1.3s · $0.009 · 331 tok
question
Trace the following Python code and give its exact output.
```python
total = 0
v = 7
while total + v <= 59:
if v % 4 != 0:
total += v
v += 6
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
39correctcode.trace.js-v1conf 99% · 1.4s · $0.009 · 308 tok
question
Evaluate the following JavaScript. What number is logged to the console? ```js const arr = [4, 5, 6, 7, 8, 9]; const out = arr .map(n => n * 7) .filter(n => n % 5 === 0) .reduce((a, b) => a + b, 0); console.log(out); ``` End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
35correctcode.trace.nested-v1conf 99% · 1.1s · $0.023 · 869 tok
question
Trace this Python program exactly. What does it print?
```python
total = 0
for i in range(1, 7):
for j in range(1, 8):
if j == 4 and i % 2 == 0:
break
if (i + j) % 3 == 0:
continue
total += i * 3 + j
print(total)
```
End your reply with exactly two plain-text lines (no markdown, no extra text after them):
ANSWER: <your final answer only>
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
260correctcode.trace.python-v1anchorconf 100% · 1.0s · $0.014 · 517 tok
model answer:
0correctcode.trace.nested-v1anchorconf 99% · 1.5s · $0.020 · 769 tok
model answer:
283correctcode.trace.js-v1anchorconf 100% · 2.0s · $0.007 · 251 tok
model answer:
63correctcode.trace.python-v1anchorconf 99% · 1.1s · $0.009 · 322 tok
model answer:
40instruction following 29/30 correct
correctif.constraints.stack-v1conf 96% · 1.8s · $0.007 · 236 tok
question
Write in English about a city at night, following ALL of these rules simultaneously: 1. Exactly 18 words. 2. The first word must be "orbit" and the last word must be "cedar". 3. Use the word "echo" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
orbit of lights glows as night sounds echo softly, each echo fading past still streets beneath tall cedarcorrectif.format.acronym-v1conf 99% · 2.3s · $0.003 · 76 tok
question
Take the third letter of each of these words, in order: comet, falcon, nova, quartz, lumen. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
MLVAMcorrectif.constraints.stack-v1conf 96% · 1.5s · $0.008 · 257 tok
question
Write in English about the sea, following ALL of these rules simultaneously: 1. Exactly 20 words. 2. The first word must be "zephyr" and the last word must be "drift". 3. Use the word "orbit" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
zephyr winds push waves into endless orbit, tides orbit softly as the moon holds each orbit near foam and driftcorrectif.format.repeat-v1conf 90% · 1.2s · $0.001 · 31 tok
question
Write the word "ember" in capitalized form, repeated exactly 3 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Ember_Ember_Embercorrectif.constraints.stack-v1conf 96% · 1.7s · $0.011 · 388 tok
question
Write in English about an old machine, following ALL of these rules simultaneously: 1. Exactly 23 words. 2. The first word must be "lumen" and the last word must be "tundra". 3. Use the word "flint" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
lumen flickered inside the ancient machine as its flint gears turned, sparking flint against flint, humming softly beneath the cold silent frozen tundracorrectif.format.acronym-v1conf 99% · 1.6s · $0.002 · 67 tok
question
Take the second letter of each of these words, in order: quartz, ember, tundra, echo. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
UMUCcorrectif.format.repeat-v1conf 99% · 1.7s · $0.003 · 99 tok
question
Write the word "nova" in capitalized form, repeated exactly 8 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
NOVA_NOVA_NOVA_NOVA_NOVA_NOVA_NOVA_NOVAcorrectif.format.acronym-v1conf 99% · 1.7s · $0.003 · 73 tok
question
Take the third letter of each of these words, in order: cedar, nova, orbit, prism, flint. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
DVBIIcorrectif.constraints.stack-v1conf 98% · 1.5s · $0.008 · 270 tok
question
Write in English about winter mornings, following ALL of these rules simultaneously: 1. Exactly 15 words. 2. The first word must be "basalt" and the last word must be "comet". 3. Use the word "tundra" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
basalt frost covers the tundra as the cold tundra waits beneath a frozen morning cometcorrectif.format.repeat-v1conf 99% · 1.1s · $0.002 · 44 tok
question
Write the word "nova" in uppercase form, repeated exactly 5 times, joined by the character "/" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
NOVA/NOVA/NOVA/NOVA/NOVAcorrectif.format.acronym-v1conf 99% · 1.5s · $0.003 · 76 tok
question
Take the second letter of each of these words, in order: falcon, basalt, nova, delta, tundra. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
AAOEUcorrectif.constraints.stack-v1conf 96% · 1.5s · $0.007 · 241 tok
question
Write in English about the sea, following ALL of these rules simultaneously: 1. Exactly 18 words. 2. The first word must be "delta" and the last word must be "flint". 3. Use the word "nova" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
delta waves roll as nova tides shine bright nova foam drifts near the nova shore beside old flintcorrectif.format.repeat-v1conf 99% · 2.2s · $0.002 · 47 tok
question
Write the word "comet" in capitalized form, repeated exactly 7 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
COMET_COMET_COMET_COMET_COMET_COMET_COMETcorrectif.format.repeat-v1conf 100% · 4.2s · $0.002 · 39 tok
question
Write the word "flint" in uppercase form, repeated exactly 4 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
FLINT_FLINT_FLINT_FLINTcorrectif.format.acronym-v1conf 99% · 1.3s · $0.003 · 74 tok
question
Take the second letter of each of these words, in order: ember, flint, cedar, comet, drift. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
MLEORcorrectif.constraints.stack-v1conf 96% · 1.7s · $0.010 · 357 tok
question
Write in English about a long journey, following ALL of these rules simultaneously: 1. Exactly 24 words. 2. The first word must be "quartz" and the last word must be "lumen". 3. Use the word "echo" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "j" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quartz cliffs faded behind as we walked west, hearing echo after echo across canyons, chasing one final echo toward that distant, glowing, warm lumencorrectif.format.acronym-v1conf 100% · 6.7s · $0.001 · 26 tok
question
Take the first letter of each of these words, in order: lumen, falcon, prism, tundra, cedar, nova. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
LFPTCNcorrectif.constraints.stack-v1conf 96% · 1.1s · $0.015 · 541 tok
question
Write in English about a city at night, following ALL of these rules simultaneously: 1. Exactly 22 words. 2. The first word must be "basalt" and the last word must be "zephyr". 3. Use the word "orbit" exactly 2 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
basalt streets shimmer as neon lights orbit above, while distant cars orbit slowly through midnight avenues, carrying dreams into the nocturnal zephyrcorrectif.format.repeat-v1conf 90% · 1.8s · $0.002 · 54 tok
question
Write the word "delta" in capitalized form, repeated exactly 7 times, joined by the character "/" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
DELTA/DELTA/DELTA/DELTA/DELTA/DELTA/DELTAcorrectif.format.acronym-v1conf 99% · 2.0s · $0.003 · 73 tok
question
Take the second letter of each of these words, in order: flint, prism, zephyr, echo, drift. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
LRECRcorrectif.constraints.stack-v1conf 96% · 1.4s · $0.011 · 392 tok
question
Write in English about an old machine, following ALL of these rules simultaneously: 1. Exactly 15 words. 2. The first word must be "tundra" and the last word must be "prism". 3. Use the word "echo" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
tundra echo echo echo machine rumbles inside cold walls near a very dusty glass prismcorrectif.format.repeat-v1conf 99% · 1.1s · $0.002 · 55 tok
question
Write the word "lumen" in capitalized form, repeated exactly 6 times, joined by the character "_" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
LUMEN_LUMEN_LUMEN_LUMEN_LUMEN_LUMENcorrectif.constraints.stack-v1conf 96% · 1.9s · $0.018 · 676 tok
question
Write in English about a city at night, following ALL of these rules simultaneously: 1. Exactly 22 words. 2. The first word must be "prism" and the last word must be "zephyr". 3. Use the word "comet" exactly 3 times (in addition to rules 2 if they differ). 4. The letter "q" must not appear anywhere. 5. Everything entirely in lowercase. Give the text itself as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
prism lights shimmer as a comet streaks above, another comet trails behind, a lone comet fades, city hums beneath the restless zephyrcorrectif.format.acronym-v1conf 99% · 4.9s · $0.003 · 81 tok
question
Take the second letter of each of these words, in order: flint, echo, comet, drift, falcon, quartz. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
LCORAUcorrectif.format.repeat-v1conf 99% · 1.1s · $0.002 · 44 tok
question
Write the word "flint" in capitalized form, repeated exactly 5 times, joined by the character "/" with no spaces. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
FLINT/FLINT/FLINT/FLINT/FLINTcorrectif.format.acronym-v1conf 99% · 2.2s · $0.002 · 65 tok
question
Take the second letter of each of these words, in order: nova, cedar, prism, basalt. Concatenate them in uppercase into a single string with no separators. Output that string as your answer. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
OERAwrongif.constraints.stack-v1anchorconf 95% · 2.9s · $0.008 · 288 tok
model answer:
drift through streets where each lumen glows softly as a second lumen flickers above us and one final lumen traces the skyline into endless orbitcorrectif.format.repeat-v1anchorconf 99% · 1.3s · $0.002 · 59 tok
model answer:
ECHO_ECHO_ECHO_ECHO_ECHO_ECHO_ECHO_ECHOcorrectif.format.repeat-v1anchorconf 99% · 1.6s · $0.002 · 67 tok
model answer:
ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYR/ZEPHYRcorrectif.format.acronym-v1anchorconf 100% · 1.2s · $0.001 · 26 tok
model answer:
ZDFQknowledge 30/30 correct
correctknowledge.fr.factbank-v2conf 100% · 1.7s · $0.001 · 25 tok
question
Name the chemical element with symbol W. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Tungstencorrectknowledge.fr.factbank-v2conf 100% · 1.8s · $0.001 · 24 tok
question
Name the Brazilian capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brasíliacorrectknowledge.fr.factbank-v2conf 100% · 1.3s · $0.001 · 24 tok
question
What is the chemical element with symbol K? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Potassiumcorrectknowledge.fr.factbank-v2conf 99% · 942ms · $0.001 · 30 tok
question
Name the author of "Snow Country". Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Yasunari Kawabatacorrectknowledge.fr.factbank-v2conf 98% · 1.1s · $0.001 · 30 tok
question
What is the author of "Snow Country"? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Yasunari Kawabatacorrectknowledge.fr.factbank-v2conf 100% · 1.7s · $0.001 · 24 tok
question
What is the element whose symbol is Sb? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Antimonycorrectknowledge.fr.factbank-v2conf 100% · 1.4s · $0.001 · 23 tok
question
Identify the chemical element with symbol Sn. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Tincorrectknowledge.fr.factbank-v2conf 100% · 1.3s · $0.001 · 23 tok
question
Name the capital of Switzerland. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Berncorrectknowledge.fr.factbank-v2conf 100% · 1.6s · $0.001 · 24 tok
question
Name the chemical element with symbol Sb. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Antimonycorrectknowledge.fr.factbank-v2conf 100% · 1.1s · $0.001 · 24 tok
question
Name the Brazilian capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brasíliacorrectknowledge.fr.factbank-v2conf 100% · 1.1s · $0.001 · 24 tok
question
Name the capital of Brazil. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brasíliacorrectknowledge.fr.factbank-v2conf 100% · 1.2s · $0.001 · 23 tok
question
Name the element whose symbol is Pb. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Leadcorrectknowledge.fr.factbank-v2conf 100% · 1.0s · $0.001 · 31 tok
question
Identify the writer of the novel "The Master and Margarita". Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Mikhail Bulgakovcorrectknowledge.fr.factbank-v2conf 100% · 2.5s · $0.001 · 23 tok
question
Identify the chemical element with symbol Pb. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Leadcorrectknowledge.fr.factbank-v2conf 100% · 964ms · $0.001 · 23 tok
question
Identify the chemical element with symbol Pb. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Leadcorrectknowledge.fr.factbank-v2conf 98% · 957ms · $0.001 · 23 tok
question
Identify the Kazakh capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Astanacorrectknowledge.fr.factbank-v2conf 98% · 1.5s · $0.001 · 23 tok
question
Identify the Kazakh capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Astanacorrectknowledge.fr.factbank-v2conf 95% · 3.1s · $0.001 · 27 tok
question
Name the Burmese capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Naypyidawcorrectknowledge.fr.factbank-v2conf 97% · 995ms · $0.001 · 27 tok
question
Name the Burmese capital city. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Naypyidawcorrectknowledge.fr.factbank-v2conf 100% · 2.0s · $0.001 · 23 tok
question
What is the element whose symbol is Sn? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Tincorrectknowledge.fr.factbank-v2conf 98% · 826ms · $0.001 · 27 tok
question
What is the capital of Myanmar? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Naypyidawcorrectknowledge.fr.factbank-v2conf 100% · 1.6s · $0.001 · 24 tok
question
Identify the chemical element with symbol K. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Potassiumcorrectknowledge.fr.factbank-v2conf 100% · 1.8s · $0.001 · 24 tok
question
What is the chemical element with symbol Hg? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Mercurycorrectknowledge.fr.factbank-v2anchorconf 100% · 5.7s · $0.001 · 24 tok
model answer:
Mercurycorrectknowledge.fr.factbank-v2conf 100% · 990ms · $0.001 · 24 tok
question
What is the capital of Brazil? Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brasíliacorrectknowledge.fr.factbank-v2conf 100% · 2.5s · $0.001 · 23 tok
question
Name the chemical element with symbol Sn. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Tincorrectknowledge.fr.factbank-v2conf 100% · 1.1s · $0.001 · 25 tok
question
Name the capital of Canada. Answer with the name only — no explanation. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Ottawacorrectknowledge.fr.factbank-v2anchorconf 100% · 1.2s · $0.001 · 25 tok
model answer:
Tungstencorrectknowledge.fr.factbank-v2anchorconf 100% · 1.6s · $0.001 · 23 tok
model answer:
Leadcorrectknowledge.fr.factbank-v2anchorconf 100% · 1.1s · $0.001 · 24 tok
model answer:
Antimonymath 30/30 correct
correctmath.counterfactual.base-v1conf 98% · 1.0s · $0.018 · 702 tok
question
Work strictly in base 7. Add the base-7 numbers 5510 and 5100. Give the result IN BASE 7. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
13610correctmath.chained.pipeline-v1conf 99% · 1.1s · $0.005 · 158 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 76 × 38. Step 2: Q = P × 7 − 417. Step 3: divide Q by 8: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
2481correctmath.percent.chain-v2conf 98% · 1.8s · $0.004 · 117 tok
question
An inventory starts at 73000 units. The warehouse was painted 9 years ago. In the first month the inventory grows by 34%. The warehouse was painted 69 years ago. The next month it shrinks by 38%, and the month after it grows by 9%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
66106.76correctmath.algebra.system-v2conf 100% · 1.1s · $0.004 · 129 tok
question
Solve the system, then answer the derived question. 4x + 3y = 67 4x − 8y = -76 What is the value of 5x − 4y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-17correctmath.counterfactual.base-v1conf 99% · 1.1s · $0.014 · 536 tok
question
Work strictly in base 11. Add the base-11 numbers 214A and 973. Give the result IN BASE 11 (digits beyond 9 are A, B, C). End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
3012correctmath.arith.chain-v2conf 99% · 2.0s · $0.003 · 113 tok
question
Calculate the following. Show your reasoning, then answer. (((49 × 53 − 988) × 3 + 4384) − 77 × 61) × 3 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
13542correctmath.chained.pipeline-v1conf 99% · 1.7s · $0.005 · 146 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 12 × 15. Step 2: Q = P × 4 − 316. Step 3: divide Q by 6: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
69correctmath.percent.chain-v2conf 98% · 1.2s · $0.004 · 107 tok
question
An inventory starts at 54000 units. The warehouse was painted 67 years ago. In the first month the inventory grows by 40%. The warehouse was painted 66 years ago. The next month it shrinks by 42%, and the month after it grows by 38%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
60510.24correctmath.algebra.system-v2conf 99% · 1.4s · $0.005 · 185 tok
question
Solve the system, then answer the derived question. 4x + 4y = -56 3x − 9y = 234 What is the value of 5x − 2y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
91correctmath.arith.chain-v2conf 99% · 1.4s · $0.003 · 114 tok
question
Evaluate the expression below and give the result. (((28 × 23 − 315) × 8 + 2604) − 24 × 79) × 3 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
10020correctmath.counterfactual.base-v1conf 100% · 1.3s · $0.009 · 325 tok
question
Work strictly in base 8. Add the base-8 numbers 5023 and 1211. Give the result IN BASE 8. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
6234correctmath.chained.pipeline-v1conf 99% · 1.2s · $0.005 · 150 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 84 × 53. Step 2: Q = P × 4 − 657. Step 3: divide Q by 7: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
2451correctmath.percent.chain-v2conf 98% · 1.8s · $0.004 · 117 tok
question
An inventory starts at 14000 units. The company was founded 55 kilometers from the port. In the first month the inventory grows by 42%. The company was founded 80 kilometers from the port. The next month it shrinks by 41%, and the month after it grows by 33%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
15599.84correctmath.algebra.system-v2conf 99% · 948ms · $0.006 · 213 tok
question
Solve the system, then answer the derived question. 9x + 8y = -209 4x − 5y = 198 What is the value of 4x − 4y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
164correctmath.arith.chain-v2conf 99% · 974ms · $0.004 · 150 tok
question
Work out the exact value of this expression. (((42 × 73 − 986) × 3 + 8925) − 41 × 19) × 6 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
86316correctmath.chained.pipeline-v1conf 99% · 2.4s · $0.005 · 150 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 68 × 28. Step 2: Q = P × 7 − 515. Step 3: divide Q by 6: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
2138correctmath.counterfactual.base-v1conf 99% · 1.0s · $0.010 · 379 tok
question
Work strictly in base 8. Add the base-8 numbers 3122 and 5003. Give the result IN BASE 8. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
10125correctmath.percent.chain-v2conf 99% · 1.3s · $0.004 · 115 tok
question
An inventory starts at 28000 units. The company was founded 3 kilometers from the port. In the first month the inventory grows by 30%. A rival firm shipped 73 unrelated parcels the same week. The next month it shrinks by 5%, and the month after it grows by 20%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
41496correctmath.algebra.system-v2conf 100% · 1.3s · $0.007 · 264 tok
question
Solve the system, then answer the derived question. 9x + 7y = 171 4x − 8y = -124 What is the value of 4x − 5y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-70correctmath.arith.chain-v2conf 99% · 1.2s · $0.004 · 143 tok
question
Work out the exact value of this expression. (((43 × 90 − 694) × 9 + 3368) − 35 × 42) × 2 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
60964correctmath.counterfactual.base-v1conf 99% · 1.1s · $0.013 · 499 tok
question
Work strictly in base 13. Add the base-13 numbers 78B and 1060. Give the result IN BASE 13 (digits beyond 9 are A, B, C). End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
181Bcorrectmath.chained.pipeline-v1conf 99% · 1.0s · $0.004 · 122 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 14 × 85. Step 2: Q = P × 5 − 943. Step 3: divide Q by 3: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1669correctmath.percent.chain-v2conf 98% · 1.9s · $0.004 · 118 tok
question
An inventory starts at 89000 units. The delivery van has a 159-liter fuel tank. In the first month the inventory grows by 37%. The warehouse was painted 37 years ago. The next month it shrinks by 20%, and the month after it grows by 15%. How many units remain (exact value, round to 2 decimals only if needed)? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
112175.60correctmath.algebra.system-v2conf 99% · 1.0s · $0.005 · 186 tok
question
Solve the system, then answer the derived question. 5x + 2y = 42 7x − 3y = -63 What is the value of 4x − 5y? End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-105correctmath.arith.chain-v2conf 99% · 1.4s · $0.003 · 113 tok
question
Evaluate the expression below and give the result. (((33 × 47 − 113) × 5 + 8875) − 44 × 30) × 2 End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
29490correctmath.chained.pipeline-v1conf 99% · 1.4s · $0.004 · 145 tok
question
Solve the following linked steps; each step uses the previous result. Step 1: P = 65 × 89. Step 2: Q = P × 5 − 918. Step 3: divide Q by 6: let q be the integer quotient and r the remainder. The final answer is q + r. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
4672correctmath.counterfactual.base-v1anchorconf 99% · 976ms · $0.012 · 464 tok
model answer:
11236correctmath.percent.chain-v2anchorconf 98% · 1.8s · $0.005 · 145 tok
model answer:
61896.52correctmath.algebra.system-v2anchorconf 100% · 1.6s · $0.003 · 88 tok
model answer:
87correctmath.arith.chain-v2anchorconf 99% · 1.2s · $0.003 · 105 tok
model answer:
108153multilingual 30/30 correct
correctmultilingual.wordnum-v1conf 99% · 1.2s · $0.002 · 70 tok
question
A number is written in French: « deux cent soixante-trois ». Another is written in Spanish: « ochenta y cuatro ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
347correctmultilingual.wordnum-v1conf 99% · 1.2s · $0.002 · 70 tok
question
A number is written in French: « cent cinquante-neuf ». Another is written in Spanish: « seiscientos ochenta y nueve ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-530correctmultilingual.numword-v2conf 96% · 836ms · $0.002 · 52 tok
question
Compute 57 + 109, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
cent soixante-sixcorrectmultilingual.numword-v2conf 99% · 980ms · $0.002 · 67 tok
question
Compute 369 + 279, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
seiscientos cuarenta y ochocorrectmultilingual.wordnum-v1conf 99% · 1.6s · $0.002 · 68 tok
question
A number is written in French: « sept cent vingt ». Another is written in Spanish: « trescientos setenta y dos ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
348correctmultilingual.numword-v2conf 96% · 1.1s · $0.002 · 52 tok
question
Compute 393 + 421, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
huit cent quatorzecorrectmultilingual.wordnum-v1conf 99% · 1.0s · $0.003 · 72 tok
question
A number is written in French: « trois cent cinquante-deux ». Another is written in Spanish: « ochocientos treinta y nueve ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1191correctmultilingual.numword-v2conf 98% · 1.1s · $0.002 · 46 tok
question
Compute 471 + 442, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
novecientos trececorrectmultilingual.wordnum-v1conf 99% · 1.9s · $0.002 · 69 tok
question
A number is written in French: « cinq cent soixante-treize ». Another is written in Spanish: « trescientos setenta y ocho ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
195correctmultilingual.numword-v2conf 96% · 1.0s · $0.002 · 58 tok
question
Compute 116 + 156, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
deux cent soixante-douzecorrectmultilingual.wordnum-v1conf 99% · 1.2s · $0.003 · 80 tok
question
A number is written in French: « huit cent quatre-vingt-douze ». Another is written in Spanish: « doscientos dieciséis ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1108correctmultilingual.numword-v2conf 98% · 1.1s · $0.002 · 65 tok
question
Compute 414 + 455, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
ochocientos sesenta y nuevecorrectmultilingual.wordnum-v1conf 99% · 1.1s · $0.002 · 70 tok
question
A number is written in French: « neuf cent quarante-six ». Another is written in Spanish: « quinientos sesenta y tres ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1509correctmultilingual.numword-v2conf 98% · 998ms · $0.002 · 63 tok
question
Compute 194 + 340, then write the result out in Spanish number words (lowercase). Answer with the Spanish words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quinientos treinta y cuatrocorrectmultilingual.wordnum-v1conf 99% · 982ms · $0.002 · 67 tok
question
A number is written in French: « soixante et un ». Another is written in Spanish: « quinientos treinta y cinco ». Compute (French number) − (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
-474correctmultilingual.numword-v2conf 96% · 1.4s · $0.002 · 56 tok
question
Compute 296 + 136, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quatre cent trente-deuxcorrectmultilingual.wordnum-v1conf 99% · 1.2s · $0.002 · 65 tok
question
A number is written in French: « six cent trente-deux ». Another is written in Spanish: « trescientos treinta y tres ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
965correctmultilingual.numword-v2conf 96% · 1.5s · $0.002 · 50 tok
question
Compute 341 + 374, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
sept cent quinzecorrectmultilingual.wordnum-v1conf 99% · 861ms · $0.002 · 69 tok
question
A number is written in French: « sept cent quarante-huit ». Another is written in Spanish: « novecientos treinta y siete ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1685correctmultilingual.numword-v2conf 96% · 1.2s · $0.002 · 60 tok
question
Compute 338 + 127, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quatre cent soixante-cinqcorrectmultilingual.wordnum-v1conf 99% · 861ms · $0.002 · 69 tok
question
A number is written in French: « neuf cent soixante-cinq ». Another is written in Spanish: « quinientos cuatro ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1469correctmultilingual.numword-v2conf 96% · 1.5s · $0.002 · 58 tok
question
Compute 294 + 430, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
sept cent vingt-quatrecorrectmultilingual.wordnum-v1conf 99% · 1.0s · $0.002 · 62 tok
question
A number is written in French: « huit cent quatorze ». Another is written in Spanish: « setecientos sesenta ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
1574correctmultilingual.numword-v2conf 90% · 1.1s · $0.002 · 54 tok
question
Compute 384 + 349, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
sept cent trente-troiscorrectmultilingual.wordnum-v1conf 99% · 1.1s · $0.003 · 84 tok
question
A number is written in French: « trois cent quatre-vingt-quatorze ». Another is written in Spanish: « cuatrocientos ». Compute (French number) + (Spanish number). Answer with digits only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
794correctmultilingual.numword-v2conf 96% · 1.1s · $0.002 · 60 tok
question
Compute 125 + 299, then write the result out in French number words (lowercase). Answer with the French words only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
quatre cent vingt-quatrecorrectmultilingual.numword-v2anchorconf 96% · 1.3s · $0.002 · 64 tok
model answer:
huit cent soixante-dix-neufcorrectmultilingual.wordnum-v1anchorconf 99% · 977ms · $0.003 · 71 tok
model answer:
150correctmultilingual.numword-v2anchorconf 98% · 1.4s · $0.002 · 57 tok
model answer:
seiscientos ochocorrectmultilingual.wordnum-v1anchorconf 99% · 1.1s · $0.003 · 71 tok
model answer:
762reasoning 28/30 correct
correctreasoning.deduction.position-v1conf 99% · 8.0s · $0.003 · 96 tok
question
Four people stand in a queue (number 1 is the front). Ines is directly ahead of Liam. Quinn is number 2 in the queue. Mona is directly ahead of Quinn. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Inescorrectreasoning.deduction.order-v2conf 98% · 1.2s · $0.010 · 359 tok
question
Seven people are ranked by who is taller (rank 1 = tallest). Ola is taller than Kira. Goran is taller than Ola. Tessa is taller than Ines. Ola is taller than Quinn. Tessa is taller than Goran. Ines is taller than Emil. Ola is taller than Quinn. Farah is faster than everyone here, but Farah is not being ranked. Emil is taller than Goran. Kira is taller than Quinn. Who is fourth (rank 4)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Gorancorrectreasoning.deduction.position-v1conf 98% · 1.7s · $0.002 · 55 tok
question
Four people stand in a queue (number 1 is the front). Ola is directly ahead of Tessa. Tessa is number 2 in the queue. Liam is directly ahead of Sami. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olacorrectreasoning.deduction.order-v2conf 98% · 1.1s · $0.007 · 242 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Ola is older than Bruno. Nadir is faster than everyone here, but Nadir is not being ranked. Ines is older than Tessa. Rosa is older than Dara. Jonas is older than Dara. Rosa is older than Jonas. Tessa is older than Rosa. Dara is older than Ola. Rosa is older than Dara. Rosa is older than Dara. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Rosacorrectreasoning.deduction.order-v2conf 99% · 1.3s · $0.010 · 361 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Rosa is faster than Kira. Goran is faster than Rosa. Kira is faster than Liam. Quinn is faster than Sami. Liam is faster than Farah. Kira is faster than Quinn. Sami is faster than Farah. Kira is faster than Farah. Emil is heavier than everyone here, but Emil is not being ranked. Sami is faster than Liam. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Kiracorrectreasoning.deduction.position-v1conf 100% · 2.4s · $0.001 · 24 tok
question
Four people stand in a queue (number 1 is the front). Rosa is number 1 in the queue. Ola is directly ahead of Goran. Goran is directly ahead of Quinn. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Rosacorrectreasoning.deduction.order-v2conf 99% · 1.2s · $0.011 · 401 tok
question
Seven people are ranked by who is taller (rank 1 = tallest). Quinn is taller than Liam. Rosa is heavier than everyone here, but Rosa is not being ranked. Tessa is taller than Quinn. Bruno is taller than Priya. Liam is taller than Emil. Bruno is taller than Tessa. Kira is taller than Bruno. Kira is taller than Tessa. Emil is taller than Priya. Liam is taller than Priya. Who is sixth (rank 6)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Emilcorrectreasoning.deduction.position-v1conf 99% · 2.7s · $0.002 · 32 tok
question
Four people stand in a queue (number 1 is the front). Dara is number 3 in the queue. Quinn is directly ahead of Dara. Nadir is directly ahead of Quinn. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Daracorrectreasoning.deduction.order-v2conf 99% · 985ms · $0.012 · 434 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Nadir is older than Quinn. Kira is older than Nadir. Sami is older than Kira. Quinn is older than Chen. Tessa is taller than everyone here, but Tessa is not being ranked. Ines is older than Sami. Dara is older than Sami. Ines is older than Dara. Kira is older than Quinn. Ines is older than Quinn. Who is sixth (rank 6)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Quinnwrongreasoning.deduction.position-v1conf 60% · 2.2s · $0.003 · 100 tok
question
Four people stand in a queue (number 1 is the front). Alice is directly ahead of Chen. Goran is number 3 in the queue. Chen is directly ahead of Goran. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
The fourth person (the remaining one)correctreasoning.deduction.order-v2conf 99% · 3.6s · $0.009 · 309 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Kira is older than Quinn. Mona is older than Rosa. Mona is older than Priya. Emil is older than Kira. Kira is older than Rosa. Liam is heavier than everyone here, but Liam is not being ranked. Farah is older than Emil. Quinn is older than Mona. Rosa is older than Priya. Kira is older than Mona. Who is second (rank 2)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Emilwrongreasoning.deduction.position-v1conf 60% · 1.7s · $0.003 · 78 tok
question
Four people stand in a queue (number 1 is the front). Tessa is directly ahead of Kira. Goran is directly ahead of Tessa. Kira is number 3 in the queue. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
The fourth person (not named among Goran, Tessa, Kira)correctreasoning.deduction.order-v2conf 99% · 2.1s · $0.009 · 291 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Quinn is faster than Ola. Bruno is faster than Liam. Bruno is faster than Ola. Tessa is faster than Goran. Liam is faster than Quinn. Priya is faster than Tessa. Tessa is faster than Ola. Dara is older than everyone here, but Dara is not being ranked. Goran is faster than Bruno. Liam is faster than Ola. Who is second (rank 2)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Tessacorrectreasoning.deduction.position-v1conf 99% · 1.2s · $0.003 · 98 tok
question
Four people stand in a queue (number 1 is the front). Ines is directly ahead of Dara. Alice is number 1 in the queue. Hana is directly ahead of Ines. Who is number 3? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Inescorrectreasoning.deduction.order-v2conf 98% · 1.2s · $0.010 · 366 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Ola is older than Farah. Farah is older than Kira. Nadir is older than Hana. Hana is older than Mona. Ola is older than Goran. Hana is older than Ola. Sami is taller than everyone here, but Sami is not being ranked. Kira is older than Goran. Mona is older than Ola. Farah is older than Goran. Who is fifth (rank 5)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Farahcorrectreasoning.deduction.position-v1conf 100% · 1.5s · $0.003 · 88 tok
question
Four people stand in a queue (number 1 is the front). Emil is directly ahead of Priya. Kira is number 4 in the queue. Priya is directly ahead of Kira. Ola is directly ahead of Emil. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olacorrectreasoning.deduction.order-v2conf 98% · 1.2s · $0.012 · 408 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Ola is heavier than Goran. Goran is heavier than Chen. Bruno is heavier than Chen. Ines is faster than everyone here, but Ines is not being ranked. Priya is heavier than Alice. Goran is heavier than Jonas. Alice is heavier than Ola. Jonas is heavier than Chen. Alice is heavier than Jonas. Jonas is heavier than Bruno. Who is second (rank 2)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Alicecorrectreasoning.deduction.position-v1conf 98% · 1.1s · $0.003 · 97 tok
question
Four people stand in a queue (number 1 is the front). Alice is directly ahead of Mona. Ola is directly ahead of Alice. Tessa is number 1 in the queue. Who is number 2? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olacorrectreasoning.deduction.order-v2conf 99% · 1.2s · $0.012 · 449 tok
question
Seven people are ranked by who is older (rank 1 = oldest). Dara is older than Goran. Ola is faster than everyone here, but Ola is not being ranked. Ines is older than Goran. Ines is older than Dara. Emil is older than Mona. Alice is older than Goran. Mona is older than Ines. Dara is older than Alice. Emil is older than Tessa. Goran is older than Tessa. Who is sixth (rank 6)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Gorancorrectreasoning.deduction.position-v1conf 99% · 1.3s · $0.002 · 65 tok
question
Four people stand in a queue (number 1 is the front). Alice is number 3 in the queue. Ines is directly ahead of Alice. Jonas is directly ahead of Ines. Who is number 1? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Jonascorrectreasoning.deduction.order-v2conf 97% · 1.7s · $0.009 · 324 tok
question
Seven people are ranked by who is heavier (rank 1 = heaviest). Goran is heavier than Emil. Chen is heavier than Alice. Mona is heavier than Alice. Ola is heavier than Chen. Chen is heavier than Alice. Emil is heavier than Chen. Dara is taller than everyone here, but Dara is not being ranked. Chen is heavier than Mona. Ola is heavier than Goran. Rosa is heavier than Ola. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Gorancorrectreasoning.deduction.position-v1conf 99% · 1.9s · $0.003 · 91 tok
question
Four people stand in a queue (number 1 is the front). Farah is directly ahead of Emil. Kira is directly ahead of Hana. Hana is number 2 in the queue. Who is number 4? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Emilcorrectreasoning.deduction.order-v2conf 98% · 995ms · $0.012 · 437 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Farah is faster than Ines. Bruno is older than everyone here, but Bruno is not being ranked. Liam is faster than Mona. Mona is faster than Ines. Mona is faster than Farah. Tessa is faster than Farah. Sami is faster than Liam. Kira is faster than Tessa. Tessa is faster than Sami. Mona is faster than Ines. Who is sixth (rank 6)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Farahcorrectreasoning.deduction.position-v1conf 99% · 1.1s · $0.003 · 108 tok
question
Four people stand in a queue (number 1 is the front). Jonas is directly ahead of Ola. Liam is directly ahead of Mona. Ola is number 2 in the queue. Who is number 2? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Olacorrectreasoning.deduction.order-v2conf 99% · 1.8s · $0.011 · 378 tok
question
Seven people are ranked by who is faster (rank 1 = fastest). Alice is faster than Liam. Liam is faster than Bruno. Priya is heavier than everyone here, but Priya is not being ranked. Bruno is faster than Kira. Chen is faster than Alice. Kira is faster than Goran. Dara is faster than Alice. Liam is faster than Goran. Chen is faster than Kira. Dara is faster than Chen. Who is third (rank 3)? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Alicecorrectreasoning.deduction.position-v1conf 99% · 1.2s · $0.003 · 85 tok
question
Four people stand in a queue (number 1 is the front). Bruno is number 2 in the queue. Farah is directly ahead of Bruno. Ines is directly ahead of Tessa. Who is number 2? Answer with the name only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
Brunocorrectreasoning.deduction.position-v1anchorconf 99% · 1.1s · $0.003 · 77 tok
model answer:
Quinncorrectreasoning.deduction.order-v2anchorconf 98% · 949ms · $0.010 · 327 tok
model answer:
Quinncorrectreasoning.deduction.order-v2anchorconf 98% · 1.0s · $0.011 · 384 tok
model answer:
Monacorrectreasoning.deduction.position-v1anchorconf 99% · 1.5s · $0.003 · 91 tok
model answer:
Farahterminal 29/30 correct
correctterminal.fs.tree-v1conf 99% · 1.0s · $0.021 · 749 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/conf`, `/proj/build`, `/proj/logs`): ``` /proj/build/draft.txt /proj/build/util.log /proj/logs/index.txt /proj/main.txt /proj/report.txt ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh mv logs/index.txt build/ mv build/util.log build/main-5.md rm report.txt cp build/draft.txt ./ rm build/index.txt cd build touch ../../proj/conf/draft-4.txt rm ../../proj/conf/draft-4.txt touch index-2.md ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/build/draft.txt
/proj/build/index-2.md
/proj/build/main-5.md
/proj/draft.txt
/proj/main.txtcorrectterminal.exit.chain-v1conf 100% · 814ms · $0.003 · 30 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt data.txt notes.txt ``` `notes.txt` contains exactly the words: coral, amber (one per line). No other files exist. These statements run in order: ```sh test -f ghost.txt && echo A || echo B test -f data.txt && echo C || echo D grep -q coral notes.txt && echo E || echo F grep -q dune notes.txt && echo G || echo H test -f ghost.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
B
C
E
H
exit:1correctterminal.pipeline.predict-v1conf 99% · 1.3s · $0.007 · 194 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score): ``` lou,hr,81,25 eli,eng,10,53 ivy,legal,6,94 fay,sales,93,47 jon,eng,14,83 bo,hr,113,95 kim,hr,61,46 gus,hr,35,10 hal,hr,49,15 ``` What is the EXACT stdout of this command? ```sh grep -F ',hr,' people.csv | cut -d, -f1,3 | sort | head -n 2 ``` Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
bo,113
gus,35correctterminal.fs.tree-v1conf 98% · 1.3s · $0.022 · 793 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/assets`, `/proj/conf`, `/proj/src`): ``` /proj/assets/notes.log /proj/assets/todo.cfg /proj/conf/setup.cfg /proj/draft.md /proj/report.cfg ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh mkdir -p assets/docs-4 rm report.cfg mv assets/todo.cfg assets/docs-4/ cd . mkdir -p logs-6 mkdir -p logs-6/assets-9 cd . touch assets/main-8.cfg touch assets/docs-4/todo-5.log cp assets/main-8.cfg logs-6/ ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/docs-4/todo-5.log
/proj/assets/docs-4/todo.cfg
/proj/assets/main-8.cfg
/proj/assets/notes.log
/proj/conf/setup.cfg
/proj/draft.md
/proj/logs-6/main-8.cfgcorrectterminal.exit.chain-v1conf 99% · 1.1s · $0.003 · 31 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt data.txt notes.txt ``` `notes.txt` contains exactly the words: dune, basil (one per line). No other files exist. These statements run in order: ```sh test -f app.txt && echo A || echo B test -f data.txt && echo C || echo D grep -q coral notes.txt && echo E || echo F test -f app.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
C
F
Z
exit:0correctterminal.fs.tree-v1conf 96% · 1.4s · $0.025 · 919 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/build`, `/proj/assets`, `/proj/logs`): ``` /proj/assets/draft.txt /proj/build/report.log /proj/index.log /proj/logs/setup.log /proj/main.log ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh mv assets/draft.txt build/ cd assets touch todo-4.cfg cp ../../proj/main.log ../../proj/logs/ touch ../../proj/notes-3.md cd . touch ../../proj/main-8.txt mv ../../proj/main.log ../../proj/todo-9.md cd ../../proj/logs ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/todo-4.cfg
/proj/build/draft.txt
/proj/build/report.log
/proj/index.log
/proj/logs/main.log
/proj/logs/setup.log
/proj/main-8.txt
/proj/notes-3.md
/proj/todo-9.mdcorrectterminal.pipeline.predict-v1conf 100% · 1.7s · $0.004 · 69 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score):
```
kim,hr,31,96
pam,sales,82,63
eli,hr,70,79
fay,eng,83,25
oli,legal,20,32
hal,legal,8,69
dev,legal,34,78
lou,hr,94,80
ana,hr,89,30
cy,legal,96,25
gus,hr,62,10
jon,legal,103,85
ivy,hr,94,73
```
What is the EXACT stdout of this command?
```sh
grep -F ',legal,' people.csv | awk -F, '{ s += $3 } END { print s }'
```
Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
261correctterminal.exit.chain-v1conf 99% · 1.3s · $0.003 · 30 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt notes.txt ``` `notes.txt` contains exactly the words: dune, amber (one per line). No other files exist. These statements run in order: ```sh true && echo A || echo B grep -q coral notes.txt && echo C || echo D test -f app.txt && echo E || echo F test -f app.txt && echo G || echo H test -f ghost.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
D
E
G
exit:1correctterminal.exit.chain-v1conf 99% · 4.8s · $0.006 · 191 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt data.txt notes.txt ``` `notes.txt` contains exactly the words: basil, amber (one per line). No other files exist. These statements run in order: ```sh test -f app.txt && echo A || echo B true && echo C || echo D test -f data.txt && echo E || echo F test -f app.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
C
E
Z
exit:0correctterminal.pipeline.predict-v1conf 99% · 982ms · $0.005 · 117 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score): ``` ned,sales,80,41 eli,legal,93,54 bo,ops,38,33 cy,sales,89,93 max,sales,119,82 kim,ops,5,30 pam,sales,40,72 dev,eng,64,29 lou,ops,79,78 jon,legal,118,30 ``` What is the EXACT stdout of this command? ```sh grep -F ',legal,' people.csv | cut -d, -f1,3 | sort | head -n 2 ``` Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
eli,93
jon,118correctterminal.fs.tree-v1conf 98% · 1.1s · $0.013 · 447 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/assets`, `/proj/logs`, `/proj/src`): ``` /proj/assets/todo.log /proj/index.md /proj/logs/draft.cfg /proj/notes.cfg /proj/src/report.md ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh rm logs/draft.cfg rm notes.cfg cp src/report.md logs/ mv index.md logs/ cp assets/todo.log ./ mkdir -p src/build-1 mv logs/report.md logs/util-3.cfg rm src/report.md ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/todo.log
/proj/logs/index.md
/proj/logs/util-3.cfg
/proj/todo.logcorrectterminal.pipeline.predict-v1conf 99% · 1.6s · $0.008 · 238 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score): ``` ana,ops,43,62 jon,eng,67,21 hal,legal,80,64 bo,ops,46,59 pam,eng,94,95 fay,eng,115,92 max,hr,75,77 oli,hr,5,36 dev,hr,62,72 gus,eng,72,90 cy,eng,42,18 lou,eng,19,53 eli,eng,10,96 ivy,eng,64,73 ``` What is the EXACT stdout of this command? ```sh grep -F ',eng,' people.csv | cut -d, -f1,3 | sort | head -n 2 ``` Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
cy,42
eli,10correctterminal.fs.tree-v1conf 98% · 2.6s · $0.017 · 610 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/docs`, `/proj/assets`, `/proj/src`): ``` /proj/assets/notes.log /proj/docs/setup.md /proj/docs/todo.log /proj/main.md /proj/report.cfg ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh touch src/index-2.log mv src/index-2.log docs/ rm docs/todo.log rm assets/notes.log cd docs touch ../../proj/src/todo-9.cfg cp index-2.log ../../proj/assets/ ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/index-2.log
/proj/docs/index-2.log
/proj/docs/setup.md
/proj/main.md
/proj/report.cfg
/proj/src/todo-9.cfgcorrectterminal.exit.chain-v1conf 99% · 1.1s · $0.006 · 186 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` data.txt notes.txt ``` `notes.txt` contains exactly the words: amber, basil (one per line). No other files exist. These statements run in order: ```sh test -f tmp.txt && echo A || echo B test -f ghost.txt && echo C || echo D grep -q coral notes.txt && echo E || echo F test -f data.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
B
D
F
Z
exit:0correctterminal.fs.tree-v1conf 98% · 1.8s · $0.021 · 778 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/src`, `/proj/build`, `/proj/conf`): ``` /proj/build/main.log /proj/build/report.log /proj/build/setup.cfg /proj/index.cfg /proj/util.log ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh rm index.cfg cd build cp ../../proj/util.log ./ touch ../../proj/report-6.txt cd ../../proj touch setup-7.cfg mkdir -p conf/conf-1 cd . rm build/util.log cp report-6.txt conf/ ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/build/main.log
/proj/build/report.log
/proj/build/setup.cfg
/proj/conf/report-6.txt
/proj/report-6.txt
/proj/setup-7.cfg
/proj/util.logcorrectterminal.pipeline.predict-v1conf 100% · 1.6s · $0.004 · 89 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score):
```
ned,hr,24,95
hal,legal,83,31
fay,sales,78,95
ivy,eng,48,95
ana,hr,110,21
bo,sales,53,15
dev,eng,54,98
jon,ops,68,51
cy,ops,90,39
max,hr,53,97
```
What is the EXACT stdout of this command?
```sh
grep -F ',hr,' people.csv | awk -F, '$4 > 65 { n += 1 } END { print n }'
```
Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
2correctterminal.exit.chain-v1conf 99% · 1.3s · $0.007 · 225 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt notes.txt ``` `notes.txt` contains exactly the words: dune, basil (one per line). No other files exist. These statements run in order: ```sh true && echo A || echo B false && echo C || echo D grep -q coral notes.txt && echo E || echo F test -f tmp.txt && echo G || echo H test -f ghost.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
D
F
H
exit:1correctterminal.pipeline.predict-v1conf 99% · 1.5s · $0.004 · 112 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score): ``` max,hr,79,42 dev,ops,71,82 ned,eng,92,88 cy,hr,4,37 fay,ops,49,79 eli,eng,96,60 oli,hr,115,65 lou,hr,13,28 gus,eng,46,70 ``` What is the EXACT stdout of this command? ```sh grep -F ',ops,' people.csv | cut -d, -f1,3 | sort | head -n 3 ``` Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
dev,71
fay,49correctterminal.fs.tree-v1conf 99% · 1.0s · $0.018 · 655 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/logs`, `/proj/assets`, `/proj/docs`): ``` /proj/assets/report.txt /proj/assets/util.cfg /proj/logs/draft.log /proj/setup.md /proj/todo.txt ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh touch logs/report-2.log cp assets/report.txt ./ cp logs/report-2.log docs/ rm logs/report-2.log cd docs touch ../../proj/notes-8.txt mkdir -p ../../proj/src-8 cd ../../proj/logs ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/report.txt
/proj/assets/util.cfg
/proj/docs/report-2.log
/proj/logs/draft.log
/proj/notes-8.txt
/proj/report.txt
/proj/setup.md
/proj/todo.txtcorrectterminal.exit.chain-v1conf 99% · 976ms · $0.002 · 33 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt notes.txt ``` `notes.txt` contains exactly the words: dune, basil (one per line). No other files exist. These statements run in order: ```sh true && echo A || echo B test -f app.txt && echo C || echo D false && echo E || echo F true && echo G || echo H test -f app.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
C
F
G
Z
exit:0correctterminal.fs.tree-v1conf 99% · 2.4s · $0.017 · 602 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/conf`, `/proj/logs`, `/proj/assets`): ``` /proj/draft.log /proj/logs/index.log /proj/logs/notes.md /proj/logs/todo.log /proj/report.md ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh mv logs/index.log logs/util-3.txt mv draft.log todo-2.md mkdir -p logs/src-8 touch conf/report-6.txt touch assets/index-9.txt touch assets/draft-5.txt mkdir -p build-6 ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/draft-5.txt
/proj/assets/index-9.txt
/proj/conf/report-6.txt
/proj/logs/notes.md
/proj/logs/todo.log
/proj/logs/util-3.txt
/proj/report.md
/proj/todo-2.mdcorrectterminal.pipeline.predict-v1conf 99% · 1.5s · $0.003 · 50 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score):
```
ana,ops,64,10
kim,legal,60,18
cy,hr,115,60
pam,sales,25,58
hal,sales,27,37
dev,sales,93,73
max,ops,38,75
fay,sales,118,55
gus,sales,91,65
eli,eng,11,85
oli,legal,119,71
ivy,eng,102,98
ned,hr,64,29
```
What is the EXACT stdout of this command?
```sh
grep -F ',ops,' people.csv | awk -F, '{ s += $3 } END { print s }'
```
Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
102correctterminal.exit.chain-v1conf 99% · 1.2s · $0.007 · 202 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` app.txt data.txt notes.txt ``` `notes.txt` contains exactly the words: dune, basil (one per line). No other files exist. These statements run in order: ```sh grep -q basil notes.txt && echo A || echo B test -f ghost.txt && echo C || echo D test -f tmp.txt && echo E || echo F test -f ghost.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
A
D
F
exit:1wrongterminal.fs.tree-v1conf 98% · 2.5s · $0.025 · 914 tok
question
A POSIX shell session starts in `/proj`. The tree initially contains these FILES (directories exist as implied, plus empty dirs `/proj/assets`, `/proj/build`, `/proj/docs`): ``` /proj/assets/notes.cfg /proj/assets/setup.log /proj/docs/main.cfg /proj/todo.log /proj/util.txt ``` These commands run in order (all succeed; `mv x dir/` moves into the directory; paths are relative to the CURRENT working directory, which `cd` changes): ```sh mkdir -p assets/logs-2 mv docs/main.cfg docs/draft-5.cfg cp util.txt assets/logs-2/ mv docs/draft-5.cfg ./ touch build/todo-5.cfg rm assets/setup.log cd . mkdir -p docs/build-1 cp assets/notes.cfg ./ ``` List every file (absolute paths) that exists afterwards, one per line, sorted in byte order (C locale). Do not list directories. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
/proj/assets/logs-2/util.txt
/proj/assets/notes.cfg
/proj/build/todo-5.cfg
/proj/docs/draft-5.cfg
/proj/draft-5.cfg
/proj/notes.cfg
/proj/todo.log
/proj/util.txtcorrectterminal.pipeline.predict-v1conf 100% · 1.8s · $0.003 · 51 tok
question
A POSIX shell session (LC_ALL=C). The file `people.csv` contains exactly these lines (columns: name,dept,units,score):
```
fay,sales,68,12
ana,legal,45,90
cy,hr,101,92
eli,legal,19,86
pam,hr,10,64
ned,legal,118,45
ivy,hr,28,19
lou,ops,61,40
max,sales,50,35
```
What is the EXACT stdout of this command?
```sh
grep -F ',legal,' people.csv | awk -F, '{ s += $3 } END { print s }'
```
Notes: plain `sort` compares bytes (so "100" sorts before "9"); `sort -k3,3n` compares field 3 numerically.
Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line:
CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>model answer:
182correctterminal.exit.chain-v1conf 99% · 1.3s · $0.005 · 136 tok
question
A POSIX shell session in a directory containing ONLY these files: ``` data.txt notes.txt ``` `notes.txt` contains exactly the words: amber, dune (one per line). No other files exist. These statements run in order: ```sh grep -q basil notes.txt && echo A || echo B true && echo C || echo D true && echo E || echo F true && echo G || echo H test -f data.txt && echo Z ``` Predict the terminal output: every line printed, in order, then a final line `exit:<N>` where N is the exit status of the LAST statement. Remember: `A && B || C` runs C whenever A fails (it is not a strict if/else); `test -f` succeeds only if the file exists; `grep -q` succeeds only if the word is present. Give your final answer inside ONE fenced code block (```), containing exactly the required content and nothing else. After the code block, add one plain-text line: CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
B
C
E
G
Z
exit:0correctterminal.fs.tree-v1anchorconf 98% · 5.6s · $0.017 · 593 tok
model answer:
/proj/build/setup-8.md
/proj/build/todo-4.md
/proj/docs/report-8.cfg
/proj/docs/util.log
/proj/main.log
/proj/report.cfg
/proj/src/index.cfgcorrectterminal.pipeline.predict-v1anchorconf 99% · 1.3s · $0.007 · 198 tok
model answer:
eli,eng,60,55
dev,eng,81,95
cy,eng,115,45correctterminal.exit.chain-v1anchorconf 99% · 946ms · $0.007 · 226 tok
model answer:
B
D
E
G
exit:1correctterminal.pipeline.predict-v1anchorconf 100% · 914ms · $0.004 · 94 tok
model answer:
1vision ocr 29/30 correct
correctvision.ocr.table-read-v1conf 99% · 1.4s · $0.006 · 54 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "A". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
138correctvision.ocr.code-hunt-v1conf 98% · 1.9s · $0.005 · 28 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
NMHXHYEVcorrectvision.ocr.table-read-v1conf 99% · 1.4s · $0.006 · 52 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
160correctvision.ocr.code-hunt-v1conf 98% · 1.7s · $0.005 · 27 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in BLUE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
XF7PPJUcorrectvision.ocr.table-read-v1conf 99% · 1.6s · $0.006 · 42 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the MAXIMUM of QTY over the rows whose GRADE is "D". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
84correctvision.ocr.code-hunt-v1conf 98% · 1.8s · $0.005 · 26 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in GREEN. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
CNM4F4correctvision.ocr.table-read-v1conf 98% · 1.9s · $0.005 · 22 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
52correctvision.ocr.code-hunt-v1conf 99% · 1.8s · $0.005 · 27 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in BLUE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
ACWW7VDcorrectvision.ocr.table-read-v1conf 99% · 2.1s · $0.006 · 44 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the MAXIMUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
92correctvision.ocr.code-hunt-v1conf 97% · 2.2s · $0.005 · 28 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
RWAWMFWTcorrectvision.ocr.table-read-v1conf 99% · 1.6s · $0.006 · 41 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
207correctvision.ocr.code-hunt-v1conf 99% · 1.6s · $0.005 · 26 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in RED. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
AH7NYCcorrectvision.ocr.table-read-v1conf 99% · 1.4s · $0.006 · 46 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the MAXIMUM of QTY over the rows whose GRADE is "D". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
25wrongvision.ocr.code-hunt-v1conf 45% · 2.2s · $0.006 · 43 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
RYKK3Hcorrectvision.ocr.table-read-v1conf 98% · 1.5s · $0.006 · 39 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the MAXIMUM of QTY over the rows whose GRADE is "D". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
86correctvision.ocr.table-read-v1conf 99% · 3.1s · $0.005 · 22 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "D". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
44correctvision.ocr.code-hunt-v1conf 98% · 2.1s · $0.005 · 27 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
EMJDHADPcorrectvision.ocr.code-hunt-v1conf 99% · 2.0s · $0.005 · 26 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in BLUE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
FNEDYWcorrectvision.ocr.table-read-v1conf 99% · 1.9s · $0.006 · 65 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the MAXIMUM of QTY over the rows whose GRADE is "A". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
86correctvision.ocr.code-hunt-v1conf 99% · 2.3s · $0.005 · 26 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
CA77CRFcorrectvision.ocr.table-read-v1conf 99% · 1.5s · $0.006 · 40 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
22correctvision.ocr.table-read-v1conf 99% · 2.0s · $0.006 · 41 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
64correctvision.ocr.code-hunt-v1conf 98% · 1.5s · $0.005 · 27 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in GREEN. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
C4UWHP7correctvision.ocr.code-hunt-v1conf 99% · 1.4s · $0.005 · 27 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in PURPLE. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
YNPVHDNcorrectvision.ocr.table-read-v1conf 99% · 1.4s · $0.006 · 43 tok
question
The image shows a table with columns ID, QTY, GRADE. Compute the SUM of QTY over the rows whose GRADE is "C". Answer with the number only. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
64correctvision.ocr.code-hunt-v1conf 99% · 1.8s · $0.005 · 26 tok
question
The image contains several printed codes in different colors, with noise. Transcribe EXACTLY the code printed in GREEN. It uses only digits and uppercase letters. End your reply with exactly two plain-text lines (no markdown, no extra text after them): ANSWER: <your final answer only> CONFIDENCE: <integer 0-100, how confident you are that your answer is correct>
model answer:
UP9UCCcorrectvision.ocr.table-read-v1anchorconf 99% · 2.5s · $0.006 · 39 tok
model answer:
15correctvision.ocr.code-hunt-v1anchorconf 98% · 1.6s · $0.005 · 25 tok
model answer:
VX7993Dcorrectvision.ocr.table-read-v1anchorconf 98% · 2.6s · $0.006 · 40 tok
model answer:
25correctvision.ocr.code-hunt-v1anchorconf 90% · 2.1s · $0.005 · 28 tok
model answer:
YH9E4AWPRun history
- 2026-08-05v0.2.0index_fit796
- 2026-08-05v0.2.0index_fit796
- 2026-08-05v0.2.0index_fit798
- 2026-08-05v0.2.0index_fit799
- 2026-08-05v0.2.0index_fit799
- 2026-08-05v0.2.0index_fit799
- 2026-08-05v0.2.0index_fit798
- 2026-08-05v0.2.0index_fit798
- 2026-08-05v0.2.0index_fit798
- 2026-08-05v0.2.0index_fit798
- 2026-08-05v0.2.0index_fit785
- 2026-08-05v0.2.0index_fit785
- 2026-08-05v0.2.0index_fit785
- 2026-08-05v0.2.0index_fit785
- 2026-08-05v0.2.0index_fit785
- 2026-08-05v0.2.0index_fit786
- 2026-08-05v0.2.0index_fit786
- 2026-08-05v0.2.0index_fit782
- 2026-08-05v0.2.0index_fit783
- 2026-08-05v0.2.0index_fit785