Bynilson

BUILD OS · AGENT BENCH

Measure a task. Don’t rank a vibe.

The benchmark foundation records the model, agent, repository version, prompt, context, interventions, tests, regressions, time and cost where available. There are no published comparative runs yet.

Status · methodology only

Published runs: 0. No winner, completion rate or speed claim is inferred from this task catalogue.

Controls before comparison

  1. Pin the repository commit and runtime environment.
  2. Use the same starting prompt and permitted context.
  3. Record all human interventions and tool iterations.
  4. Run the same tests and regression checks.
  5. Report failures and missing values; do not infer quality from one task.

Task families

  • feature

    • Add role-based access
    • Add a Stripe checkout flow
    • Add an onboarding flow

  • debugging

    • Repair a broken webhook
    • Diagnose an auth redirect loop
    • Resolve an RLS regression
    • Repair a failed migration

  • refactoring

    • Extract a service layer
    • Reduce proven duplication
    • Migrate an API structure

  • testing

    • Add meaningful end-to-end coverage
    • Reproduce a bug
    • Detect a regression

  • discovery

    • Diagnose canonical metadata
    • Identify a real indexing blocker

Read the agent benchmark protocol →Research plan: state of AI-built software →