New Benchmark Finds Coding Agents Struggle Badly at Oncall Root Cause Analysis
ORCA-bench tested five frontier coding agents on 1,079 oncall tasks, and the best agent reached just 25.3% accuracy on medium-difficulty cases.
What the researchers built
Large language models can write, patch, and search code, but the paper's authors argue that oncall root cause analysis, or RCA, demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. Researchers led by Albert Gong introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live, OpenTelemetry-instrumented microservice system, exposing six days of metrics, logs, and traces through real telemetry interfaces including Prometheus, Jaeger, and OpenSearch via Grafana, plus full source-code access, with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios.
How the researchers checked the results
Ground-truth symptoms were curated and signed off by expert engineers, and the paper's own LLM-as-judge scoring was independently re-scored by humans, with the paper reporting a Cohen's weighted kappa of 0.90 between the two. That level of agreement is meant to show the automated scoring closely tracked what human reviewers concluded.
What the agents got right, and wrong
Across five frontier agents, the paper reports the best RCA accuracy was 25.3% on Medium-difficulty tasks, which the authors describe as the realistic-input setting, and 10.0% on Hard tasks. The authors note that this gap remains even with Claude Fable 5, one of the agents tested. The weakest model tested hallucinated an implausible root cause in 40% of incident reports, and the paper found that removing source-code access degraded every metric it measured, meaning agents did worse across the board without the code in front of them.
Why the authors call this a lower bound
The authors are explicit that these results come from a curated 50 GB, six-day testbed, with each task investigated in isolation on a system whose code and instrumentation are public. Real production systems, the authors write, are order of magnitudes larger, more dynamic, and more idiosyncratic than ORCA-bench's testbed. Because of that gap, the authors describe their reported numbers as a lower bound on the engineering investment still required before frontier coding agents can be safely entrusted with production reliability work. The paper also notes the public dataset has been released for other researchers to use.
Key facts
- ORCA-bench pairs a live, six-day, OpenTelemetry-instrumented microservice system with 1,079 root cause analysis tasks.
- Across five frontier agents, the best RCA accuracy was 25.3% on Medium-difficulty tasks and 10.0% on Hard tasks.
- The weakest agent hallucinated an implausible root cause in 40% of incident reports.
- Removing source-code access degraded every metric the researchers measured.
- The paper reports a Cohen's weighted kappa of 0.90 between its automated LLM-as-judge scoring and independent human re-scoring.
Check your understanding
- What accuracy did the best agent reach on Medium-difficulty tasks, and how did that compare with Hard tasks?
- Why do the authors describe their reported results as a lower bound rather than a full picture of production readiness?