aws-bench Tests AI Agents on Real, Disposable AWS Infrastructure
aws-bench is an open-source benchmark that grades AI agents on live AWS tasks inside disposable cloud accounts, then scores every attempt as a pass or a fail.
What aws-bench does
aws-bench measures how well AI agents and model combinations, such as Claude Code with Sonnet or Codex with GPT, perform real AWS work: diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. According to its GitHub repository, the project scores agents against disposable, real AWS environments instead of static fixtures, so the results reflect the kind of work AWS practitioners actually do.
How a run works
Each run provisions an isolated AWS account and deploys a scenario built from real infrastructure defined as CDK stacks. The agent then attempts a task inside a sandboxed container with scoped AWS credentials. An automated verifier scores the outcome: an LLM judge checks read-only diagnosis tasks by comparing the agent's answer against a reference answer, while a programmatic check inspects live AWS state for tasks that create or modify resources, then rolls the changes back. Every trial writes a reward of 1.0 for a pass or 0.0 for a fail into a results folder.
Built on Harbor
aws-bench extends Harbor, an existing open-source project for evaluating AI agents and language models, adding the AWS-specific environment provisioning, scenarios, and verifiers described above. Datasets of tasks and their scenarios live in a companion repository, aws-bench-datasets, and every scenario can also run as its own standalone dataset, so a team can work through the benchmark one environment at a time instead of running the full suite at once.
What you need to try it
Running aws-bench requires Docker with the Compose v2 plugin and buildx version 0.17.0 or later, plus an AWS account with permission to create an AWS Organization and member accounts, since aws-bench provisions its own disposable test accounts under that organization. The tool expects the us-east-1 region for its own operations. Model access is flexible: agents can call any large-language-model provider, and aws-bench's own credential helper only matters if a team chooses to run models on Amazon Bedrock.
Key facts
- aws-bench scores agents against disposable, real AWS accounts rather than static fixtures.
- Tasks fall into two families: read-only introspection, judged by an LLM against a reference answer, and mutation tasks, checked against live AWS state and then rolled back.
- Every trial records a reward of 1.0 for a pass or 0.0 for a fail.
- The project is built on Harbor, an existing open-source agent-evaluation project, and requires Docker Compose v2 plus buildx 0.17.0 or later.
- Scenarios include areas such as ec2-multiregion, databases-and-storage, serverless-apps, and troubleshooting-multiservice, and each can run as its own standalone dataset.
The basic workflow
A typical session runs three commands in order: aws-bench env init and env setup to provision the disposable accounts and deploy a scenario's infrastructure, aws-bench run with flags for the dataset, environment name, agent, and model to execute the benchmark, and aws-bench env cleanup plus env terminate to remove the deployed resources and close the test accounts afterward. Before any of that, a team sets AWS_REGION=us-east-1 and AWS_DEFAULT_REGION=us-east-1 in its shell so its own AWS command-line and SDK calls resolve to the same region aws-bench expects.
Check your understanding
- What two ways does aws-bench score an agent's attempt at a task, and when is each one used?
- What values does a trial's reward.json record for a pass versus a fail?
- Which existing open-source project does aws-bench extend, and what does aws-bench add on top of it?