HumaneCircle benchmarks
Measuring trust, not just safety
AI systems now act for people: they approve refunds, share records, send payments and call tools. Whether they should is a trust decision, and most benchmarks do not measure it. We build benchmarks that do, starting with RealTrustDecision.
How to think about trust
Trust is a decision, not a filter
A safety filter asks whether a piece of text is harmful. A trust decision asks whether this action, by this system, for this person, may go ahead now, and on what terms. Getting that right takes more than spotting bad content.
The answer depends on the rules
The same refund, data request or payment can be fine under one policy or jurisdiction and wrong under another. When the rules are known, the decision should follow them; when they are not, a careful system says so.
More than yes or no
Real decisions allow an action with a safeguard, hold it until a missing fact arrives, or hand it to a person. A system that can only allow or block will block too much or allow what should have waited.
Some errors cost more
Approving what should have stopped is the expensive mistake. It has to be measured on its own, not averaged away with over-caution.
Grounded in real outcomes
The right answer should come from how real authorities decided, not from what annotators guess. That is the standard a regulator, an auditor or a customer will hold the system to.
RealTrustDecision
Real decisions, made by real authorities
Our first benchmark, in two parts: RTD-Full and RTD-Rules.
AI systems are moving from answering questions to taking actions: approving a refund, sharing a record, sending a payment, calling a tool on someone's behalf. Each of those is a trust decision: may this go ahead, and on what terms? Safety benchmarks ask a narrower question, whether a piece of text is harmful. We built RealTrustDecision to measure the decision itself.
Real rulings, not annotator labels
Every case comes from a public decision by a regulator, court, ombudsman, data-protection authority, or in a published security advisory, rewritten as the decision an AI system or agent faces.
Five outcomes, not two
Many actions may go ahead only with a safeguard, must wait for something, or need a person to decide. A system that can only allow or block will block too much or allow what should have waited.
RTD-Full and RTD-Rules
RTD-Full: decide from the situation alone, as a careful expert would. RTD-Rules: the governing rules are stated and must be applied, the way a company deploys AI under its own policies.
The costly error, on its own
False approvals (allowing what should have stopped) are scored separately from over-blocking.
Results
| Benchmark | Model | Exact | False approvals | Over-blocking |
|---|---|---|---|---|
| RTD-Rules | Humane TIM v1-fast | 77.6% | 5.2% | 5.0% |
- Exact accuracy over the five outcomes. False approvals and over-blocking are scored on their own.
- Early finding across every system tested: ESCALATE is almost never chosen. Systems rarely recognize when a person must decide.
More to come
Benchmarks for every place AI makes a decision
RealTrustDecision is the first. We are building benchmarks for the other decisions AI systems make when they act: agents choosing and using tools under policy, decisions in regulated industries such as financial services, and the handling of sensitive data. Each follows the same principles: real cases, more than two outcomes, and the costly error scored on its own.
These benchmarks measure AI systems in general. For each customer we set Humane TIM up on your rules and test it on your own scenarios, aiming for 99% on that internal benchmark, so one trust model can make the decisions across your applications and agents. Results depend on your scenarios and use cases.