Coding Agents Reliability Evals Harness Engineering

AutomationBench

AI benchmark leaderboard

An agent-automation benchmark and evaluation harness for assessing whether AI agents can complete automation tasks, including the reliability of their trajectories and verifiers.

Recent stories

1 linked story

workflowPRIMARY2026-08-03

Agent studies trace reliability failures to harness design and verifier quality

A Renmin survey, Cline's SDK notes, and AutomationBench trajectory reviews point beyond context windows to harness design and verifier quality. A failure-mode paper maps issues across models, memory, tools, users, and environment.