IBM's Retail Agent Hits 67.5% on τ²-bench Without Fine-Tuning: What Memory-Based Agents Mean for Teams
IBM Research's retail agent with learned guidelines ranks first on τ²-bench at 67.5% Pass^4 without fine-tuning. How to read the result, the split caveat, the held-out airline evidence, and what it means for teams on hosted models.
IBM Research reports that a retail customer-service agent using learned guidelines, rather than a fine-tuned model, ranks first on the τ²-bench retail leaderboard with a Pass^4 score of 67.5%. The result is interesting because it suggests teams can improve an off-the-shelf agent by giving it a learned policy document instead of retraining weights. It also comes with caveats about how the leaderboard is scored that anyone quoting the number should understand.
This article explains what IBM reported, how to read the τ²-bench figures correctly, and what the approach means for teams building agents on hosted models.
The headline result
According to Unite.AI’s October 20 report, IBM’s retail agent with learned guidelines placed first on τ²-bench at 67.5% Pass^4, ahead of RAFT-30B-A3B at 61.4%, a model IBM describes as fine-tuned on synthetic τ²-bench retail data. The margin is 6.1 points, and the leaderboard snapshot cited is dated October 12, 2026.
IBM’s framing is that its approach improves stock agents without weight updates, human annotation, or test-set leakage. In other words, the company claims to beat a fine-tuned model by mining guidelines from the agent’s own runs and adding them to its instructions.
How to read Pass^k
τ²-bench measures whether a customer-service agent completes tasks correctly while following domain policies. Pass^k is the probability that an agent succeeds on all k independent attempts at the same task, so it measures consistency rather than best-case performance.
IBM’s retail submission, filed as a τ²-bench pull request dated September 14, 2026, shows how much that matters:
| Metric (retail, 114 base-split tasks, 4 trials) | Score |
|---|---|
| Pass^1 | 81.14% |
| Pass^4 | 67.54% |
| Average cost per conversation (tau2-bench v1.0.1) | $0.1084 |
The 13.6-point drop from Pass^1 to Pass^4 is a useful reminder that single-trial results overstate reliability. Teams that evaluate their own agents with one run per task are effectively reporting Pass^1, which will look better than what users experience over repeated interactions.
The split caveat
Unite.AI notes that leaderboard submissions are scored on the full base split, which combines train and test tasks, and that IBM’s guidelines were learned from a retail train split that is included in that base split. IBM disclosed that it modified prompts and said it expects the submissions to be listed as Unverified; both pull requests were still open at the time of the report.
This does not mean the result is invalid, but it does change how it should be compared. A first-place Pass^4 on a split that includes the training tasks is not equivalent to a Pass^4 on a held-out set, and the 67.5% figure should not be placed next to held-out results from other papers as if they were measured the same way.
Airline results: a different domain and model
IBM also submitted an airline result, dated September 21, 2026:
| Metric (airline, 50 tasks) | Score |
|---|---|
| Pass^1 | 88.50% |
| Pass^4 | 82.00% |
| Average cost per conversation | $0.1690 |
IBM reports that its gpt-5.5-based agent ranks first in the airline domain at 82.0%, above frontier entries it lists at 66–72%. A gpt-4.1 variant with high-support guidelines placed eighth at 44%, ahead of GPT-5.2 running without reasoning and close to Claude Sonnet 4.5 in IBM’s comparison.
Two points are worth keeping in mind. First, airline (50 tasks) and retail (114 tasks) are different domains with different policies, so “first on τ²-bench” really means two separate results. Second, the gpt-4.1 result shows that learned guidelines do not lift every model to the top; the base model still matters a great deal. The 66–72% frontier range is IBM’s own list, so readers who need the current standings should check the live leaderboard.
The held-out comparison is the cleaner evidence
The more convincing evidence in IBM’s report is a held-out airline experiment. IBM used gpt-5.5 as both the agent and the guideline-extraction model, mining guidelines from the agent’s own runs. Because no extracted guideline reached a support of at least three (meaning it appeared in at least three runs), the high-support set used a threshold of two (“sup2”), producing 103 guidelines. A “lossless” set kept 146 guidelines.
| Configuration (held-out airline tasks) | Accuracy | Pass^4 |
|---|---|---|
| gpt-5.5 baseline | 72.5% | 55.0% |
| sup2 (103 guidelines) | 80.0% | 70.0% |
| Lossless (146 guidelines) | 82.5% | 70.0% |
IBM describes this as gains of up to 10 accuracy points and 15 Pass^4 points, and the recap says the lossless result reproduced exactly across reruns. Because the baseline and guideline versions were measured on the same held-out tasks with the same model, this comparison avoids the split problem in the leaderboard number.
The table also shows that the two guideline sets tied on Pass^4 at 70.0%; the larger lossless set improved accuracy (82.5% versus 80.0%) but not consistency. For anyone reimplementing the method, the support threshold is an important setting: it determines how many guidelines are kept and may affect results differently across domains.
What “no weight updates” means in practice
For teams using hosted models, the practical appeal is clear. Learned guidelines let you keep a vendor model pinned to a specific version while still improving behavior with a policy document derived from real interactions. That fits common operational constraints: no training infrastructure, no need to manage custom weights, and easy rollback.
The approach also introduces responsibilities:
- Treat the guideline file like a system prompt. It shapes agent behavior directly, so it should be version-controlled, reviewed, and protected from unauthorized edits.
- Review guidelines for policy risk. Guidelines mined from traces can encode behavior that scores well on a benchmark but violates business rules, such as overly generous refunds. Benchmarks do not catch every such case.
- Account for skipped human annotation. Not needing annotation saves cost, but it also means nobody has checked whether the learned guidelines handle situations the benchmark does not cover.
- Separate train and evaluation data. If you mine guidelines from production traces, evaluate on cases that were not used to generate them.
Cost
The reported costs, about $0.11 per retail conversation and $0.17 per airline conversation, are useful for rough budgeting. They are benchmark costs on tau2-bench v1.0.1, though, and production conversations with longer histories, more tools, or different models will cost differently. The higher airline cost is consistent with that domain’s longer policies.
Memory as a product category
IBM’s result arrives as several companies are positioning “learned memory” as a distinct layer of agent infrastructure. In the same week, Markets Insider carried an announcement that MemoraX AI researchers and collaborators had 10 papers accepted to NeurIPS 2026. Founder and CEO Jianye Hao argued that memory should be a learnable capability and an independent infrastructure layer rather than an extension of a foundation model. The company’s products include a Memory API (write, search, update, delete) and MemoraX Code, which is designed to retain repository structure, architectural decisions, and debugging history across sessions.
That announcement is a company statement, not a benchmark result, and it should not be compared directly with IBM’s numbers. The shared idea is that agents improve when they can accumulate and reuse experience, rather than relying only on a longer context window.
A checklist for evaluating similar claims
When a vendor or research team reports a benchmark result for an agent method, three questions help:
- Which split was scored? Does the evaluation set include the data the method learned from?
- Which Pass^k? Is consistency across multiple trials reported, or only a single run?
- What are the key settings? For guideline methods, what support threshold was used, and how many guidelines were kept?
For related coverage of benchmark reporting, see our analysis of running large models on clustered Mac Studios and byte-level language models.
Bottom line
IBM’s learned-guidelines approach is a credible way to improve hosted agents without fine-tuning, and its held-out airline experiment (Pass^4 rising from 55% to 70%) is the strongest evidence for it. The headline 67.5% retail leaderboard score is real but should be quoted with its context: a base split that includes the training tasks, an Unverified status, and a cost of about $0.11 per conversation on the benchmark.
Sources
- Unite.AI, “IBM Research reports learned memory tops τ²-bench service benchmark” (October 20): https://www.unite.ai/ibm-research-reports-learned-memory-tops-tau-2-bench-service-benchmark
- Markets Insider, MemoraX NeurIPS 2026 announcement (October 22): https://markets.businessinsider.com/news/stocks/memorax-ai-researchers-and-collaborators-have-10-papers-accepted-to-neurips-2026-1036655433