# New Benchmark Finds Coding Agents Struggle Badly at Oncall Root Cause Analysis

**ORCA-bench tested five frontier coding agents on 1,079 oncall tasks, and the best agent reached just 25.3% accuracy on medium-difficulty cases.**

## What the researchers built

Large language models can write, patch, and search code, but the paper's authors argue that oncall root cause analysis, or RCA, demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. Researchers led by Albert Gong introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live, OpenTelemetry-instrumented microservice system, exposing six days of metrics, logs, and traces through real telemetry interfaces including Prometheus, Jaeger, and OpenSearch via Grafana, plus full source-code access, with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios.

## How the researchers checked the results

Ground-truth symptoms were curated and signed off by expert engineers, and the paper's own LLM-as-judge scoring was independently re-scored by humans, with the paper reporting a Cohen's weighted kappa of 0.90 between the two. That level of agreement is meant to show the automated scoring closely tracked what human reviewers concluded.

## What the agents got right, and wrong

Across five frontier agents, the paper reports the best RCA accuracy was 25.3% on Medium-difficulty tasks, which the authors describe as the realistic-input setting, and 10.0% on Hard tasks. The authors note that this gap remains even with Claude Fable 5, one of the agents tested. The weakest model tested hallucinated an implausible root cause in 40% of incident reports, and the paper found that removing source-code access degraded every metric it measured, meaning agents did worse across the board without the code in front of them.

## Why the authors call this a lower bound

The authors are explicit that these results come from a curated 50 GB, six-day testbed, with each task investigated in isolation on a system whose code and instrumentation are public. Real production systems, the authors write, are order of magnitudes larger, more dynamic, and more idiosyncratic than ORCA-bench's testbed. Because of that gap, the authors describe their reported numbers as a lower bound on the engineering investment still required before frontier coding agents can be safely entrusted with production reliability work. The [paper](https://arxiv.org/abs/2607.28545) also notes the public dataset has been released for other researchers to use.

## Key facts

- ORCA-bench pairs a live, six-day, OpenTelemetry-instrumented microservice system with 1,079 root cause analysis tasks.
- Across five frontier agents, the best RCA accuracy was 25.3% on Medium-difficulty tasks and 10.0% on Hard tasks.
- The weakest agent hallucinated an implausible root cause in 40% of incident reports.
- Removing source-code access degraded every metric the researchers measured.
- The paper reports a Cohen's weighted kappa of 0.90 between its automated LLM-as-judge scoring and independent human re-scoring.

## Check your understanding

- What accuracy did the best agent reach on Medium-difficulty tasks, and how did that compare with Hard tasks?
- Why do the authors describe their reported results as a lower bound rather than a full picture of production readiness?

## Sources

- [ORCA-bench: How Ready Are Language Model Agents for Oncall? (arXiv)](https://arxiv.org/abs/2607.28545)

---

- Category: learn (research)
- Affects: —
- Published: 2026-07-30T00:00:00.000Z
- Original: https://arxiv.org/abs/2607.28545
- Generated by tinker news