← Agent Bench

METHOD · CURRENTLY NO RESULTS

Reproducible coding-agent runs.

Every experiment records enough context for another person to understand what ran. Missing fields remain missing; they are not estimated after the fact.

Required run record

  • taskId
  • repoVersion
  • agent
  • model
  • date
  • startingPrompt
  • allowedContext
  • constraints
  • result
  • testsPassed
  • testsFailed
  • regressions
  • iterations
  • humanInterventions
  • filesTouched

Comparison rules

  1. Pin the repository commit and runtime environment.
  2. Use the same starting prompt and permitted context.
  3. Record all human interventions and tool iterations.
  4. Run the same tests and regression checks.
  5. Report failures and missing values; do not infer quality from one task.

A single task describes one run under one setup. It cannot establish a universal claim about an agent or model.