First results coming

Which decision model can you trust to act on its own?

Decision models such as TypeSafe Jev and Cloudflare Clef answer typed questions in one pass and give a probability for every option. decidingbench is an independent benchmark that checks those answers against what really happened in real records: tool calls that were allowed or blocked, tickets that went to a team, posts that were removed. It reports how often each model is right when it is confident enough to act, how many cases that leaves it to handle, and what each decision costs and how long it takes through the public API. Part of the test set is public so anyone can rerun it. The rest is collected after the tested models were released and kept private, so no model can have trained on it.