Researchers Introduce DataClawEval, a Tougher Benchmark for Data Engineering Agents
A new preprint benchmark, DataClawEval, tests AI agents on real data engineering work, and the researchers report the strongest agent scored only 74.9.
What the researchers built
A team of seven researchers — Debin Meng, Jiaming Yang, Zefang Zong, Tengyue Xu, Haining Xie, Yang Li, and Peng Chen — posted a preprint to arXiv describing DataClawEval, which they call the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. The paper argues that existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored, which is the gap the authors say DataClawEval is meant to bridge.
How the benchmark works
According to the preprint, DataClawEval is built upon production-grade code authored by professional enterprise data engineers and comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, the authors say each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts, so a passing score depends on whether the agent's output actually behaves correctly rather than on another model's opinion of it. The researchers say they release the dataset, containerized environments, and deterministic evaluation scripts publicly for others to reuse.
What they found
The authors report evaluating 16 frontier agents on DataClawEval and say the results expose critical limitations: the strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine. The paper frames this as revealing strict domain specialization rather than omnipotent proficiency across PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL alike. The authors conclude that autonomous data engineering "remains a formidable, unresolved challenge," a claim they base on this one evaluation setup rather than on a broader industry survey.
Why the hedging matters
DataClawEval is a preprint: it was submitted to arXiv on 30 Jul 2026 and, as of this writing, has not gone through formal peer review. The 74.9 score and the 16-agent comparison are the authors' own reported results from their own sandboxes and grading scripts, not an independently verified industry benchmark. Readers should treat the numbers as what the researchers measured in this specific setup, not as a settled, universal ranking of agent capability.
Key facts
- DataClawEval is a preprint benchmark submitted to arXiv on 30 Jul 2026 by Debin Meng and six co-authors, for evaluating AI agents on end-to-end data engineering tasks.
- It comprises 100 tasks built on production-grade code from professional enterprise data engineers, spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL.
- Each task runs inside an isolated sandbox and is graded by deterministic, rule-based scripts rather than an LLM-as-a-judge.
- The researchers evaluated 16 frontier agents and report the strongest model reached only 74.9 overall, with no single model leading on every engine.
- The authors say the dataset, containerized environments, and evaluation scripts are released publicly on GitHub.
Check your understanding
- How many tasks does DataClawEval include, and how many different execution engines do they span?
- What score did the strongest of the 16 evaluated agents reach, according to the researchers, and why do the authors say no single model dominates?