Designing agents that earn trust instead of asking for it
Autonomy is not a slider you turn up. It is something a system is granted, one verifiable action at a time.
The first version of our agent runtime could do more than the current one. We removed capability on purpose, and adoption went up.
The failure mode nobody plans for
An agent that is right 92% of the time and opaque about all of it is worse than an agent that is right 80% of the time and shows its reasoning. The first one gets switched off after its first visible mistake, because nobody can tell whether that mistake was an outlier or the tip of something.
Accuracy buys you permission to run. Legibility is what lets you keep running.
Three constraints we hold
- Every action carries its evidence. Not a confidence score, but the actual rows, messages, and events that led to it, one click away.
- Reversibility before autonomy. An agent may only take an action unsupervised if a human can undo it in a single step.
- Rejections are training data. When a rep declines a suggestion, that is the most valuable signal in the system, and it is wasted if it only removes a card from a queue.
We would rather ship an agent that asks twice and is trusted for years than one that never asks and is turned off in a month.
What this costs
It is slower. Evidence trails are real engineering, reversibility constrains what you can build, and treating rejections as training data means a feedback loop you have to maintain rather than a log you can drop.
The trade is worth making because the alternative is not a faster product. The alternative is a product that gets disabled in month three, which is infinitely slower.
