
CRM workflow model · Ack 9B
Qwen3.5-9Bfine-tuned + RL90 held-out taskssecond company, held out
Verified task completion
75 / 90Ack 9B (fine-tuned + RL)
tasks completed correctly
- Haiku 5.5 + written policy61 / 90
- Ack 9B (fine-tuned + RL)75 / 90
- Haiku 5.5 + rules we extracted by hand77 / 90
Not an apples-to-apples baseline: we read the company's history and wrote the rules for Haiku by hand. That extraction is the work Ack is built to automate.
First company · 89 tasksthis test informed our data design
- Haiku 5.5 + written policy50 / 89
- Sonnet 5.5 + written policy53 / 89
- Ack 4B (fine-tuned)79 / 89
- Haiku 5.5 + rules we extracted by hand77 / 89
Not an apples-to-apples baseline: these rules were written by hand after looking at test failures.
Synthetic Salesforce CRMArena-Pro data (CC-BY-NC research data, credit Salesforce) · deterministic database grading · single training seed · no customers yet
See the results



