# aws-bench Tests AI Agents on Real, Disposable AWS Infrastructure

**aws-bench is an open-source benchmark that grades AI agents on live AWS tasks inside disposable cloud accounts, then scores every attempt as a pass or a fail.**

## What aws-bench does

aws-bench measures how well AI agents and model combinations, such as Claude Code with Sonnet or Codex with GPT, perform real AWS work: diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. According to its [GitHub repository](https://github.com/aws-bench/aws-bench), the project scores agents against disposable, real AWS environments instead of static fixtures, so the results reflect the kind of work AWS practitioners actually do.

## How a run works

Each run provisions an isolated AWS account and deploys a scenario built from real infrastructure defined as CDK stacks. The agent then attempts a task inside a sandboxed container with scoped AWS credentials. An automated verifier scores the outcome: an LLM judge checks read-only diagnosis tasks by comparing the agent's answer against a reference answer, while a programmatic check inspects live AWS state for tasks that create or modify resources, then rolls the changes back. Every trial writes a reward of 1.0 for a pass or 0.0 for a fail into a results folder.

## Built on Harbor

aws-bench extends Harbor, an existing open-source project for evaluating AI agents and language models, adding the AWS-specific environment provisioning, scenarios, and verifiers described above. Datasets of tasks and their scenarios live in a companion repository, aws-bench-datasets, and every scenario can also run as its own standalone dataset, so a team can work through the benchmark one environment at a time instead of running the full suite at once.

## What you need to try it

Running aws-bench requires Docker with the Compose v2 plugin and buildx version 0.17.0 or later, plus an AWS account with permission to create an AWS Organization and member accounts, since aws-bench provisions its own disposable test accounts under that organization. The tool expects the us-east-1 region for its own operations. Model access is flexible: agents can call any large-language-model provider, and aws-bench's own credential helper only matters if a team chooses to run models on Amazon Bedrock.

## Key facts

- aws-bench scores agents against disposable, real AWS accounts rather than static fixtures.
- Tasks fall into two families: read-only introspection, judged by an LLM against a reference answer, and mutation tasks, checked against live AWS state and then rolled back.
- Every trial records a reward of 1.0 for a pass or 0.0 for a fail.
- The project is built on Harbor, an existing open-source agent-evaluation project, and requires Docker Compose v2 plus buildx 0.17.0 or later.
- Scenarios include areas such as ec2-multiregion, databases-and-storage, serverless-apps, and troubleshooting-multiservice, and each can run as its own standalone dataset.

## The basic workflow

A typical session runs three commands in order: `aws-bench env init` and `env setup` to provision the disposable accounts and deploy a scenario's infrastructure, `aws-bench run` with flags for the dataset, environment name, agent, and model to execute the benchmark, and `aws-bench env cleanup` plus `env terminate` to remove the deployed resources and close the test accounts afterward. Before any of that, a team sets `AWS_REGION=us-east-1` and `AWS_DEFAULT_REGION=us-east-1` in its shell so its own AWS command-line and SDK calls resolve to the same region aws-bench expects.

## Check your understanding

- What two ways does aws-bench score an agent's attempt at a task, and when is each one used?
- What values does a trial's reward.json record for a pass versus a fail?
- Which existing open-source project does aws-bench extend, and what does aws-bench add on top of it?

## Sources

- [GitHub: aws-bench/aws-bench](https://github.com/aws-bench/aws-bench)

---

- Category: implement (tool)
- Affects: —
- Published: 2026-07-26T07:31:49.066Z
- Original: https://github.com/aws-bench/aws-bench
- Generated by tinker news