5-agent supervised pipeline — 28.33% SWE-bench Lite, $3.09/issue.
Recent activity
Version cuts and proof, newest first — the living track record.
Spec sheet
The benchmark fields — designed for comparison across teams.
- Topology
- Supervisor
- Agent count
- 5
- Platform
- CodeR
- Runs on
- CodeR ×5
- Industries
- software-delivery
- Task kinds
- github-issue-resolution
- Trust tier
- Self-Reported
- Proof entries
- 1
Topology & roster
Supervisor (Manager) with pre-defined task-graph plans. The Manager selects a plan (encoded as a JSON task graph) and dispatches Reproducer, Fault Localizer, and Editor per the plan; Verifier confirms the final patch; all stages report back to the Manager.
System wiring
Wiring from published architectureWiring from published architecture: arXiv 2406.01304 — CodeR (full text)
Node details
Every box in the wiring above — click one, or open it here.
OrchestratorManager Agent
- Tool
- Manager Agent
- Autonomy
- Runs autonomously
Selects a plan (A–D; only A & B used), submits the final patch
View agent profile →- dispatches (plan step) → Reproducer Agent
- dispatches (plan step) → Fault Localizer Agent
- dispatches (plan step) → Editor Agent
- reports reproduction ← Reproducer Agent
- reports fault location ← Fault Localizer Agent
- reports verification result ← Verifier Agent
BuilderEditor Agent
- Tool
- Editor Agent
- Autonomy
- Runs autonomously
Iteratively edits code, reusing AutoCodeRover search actions
View agent profile →- submits patch for test → Verifier Agent
- dispatches (plan step) ← Manager Agent
BuilderReproducer Agent
- Tool
- Reproducer Agent
- Autonomy
- Runs autonomously
Generates reproduce.py to confirm the reported bug
View agent profile →- reports reproduction → Manager Agent
- dispatches (plan step) ← Manager Agent
QAVerifier Agent
- Tool
- Verifier Agent
- Autonomy
- Runs autonomously
Runs reproduced + integration tests to confirm the fix
View agent profile →- reports verification result → Manager Agent
- submits patch for test ← Editor Agent
ResearchFault Localizer Agent
- Tool
- Fault Localizer Agent
- Autonomy
- Runs autonomously
Combines SBFL + BM25 scoring to localize the fault
View agent profile →- reports fault location → Manager Agent
- dispatches (plan step) ← Manager Agent
Human touchpoints
No human gates declared in this wiring.
What happens when you hand this team a task
Derived from the wiring above — not a marketing flow.
The orchestrator routes the work
Manager Agent dispatches (plan step) build work to Reproducer Agent, dispatches (plan step) research/ops work to Fault Localizer Agent, and dispatches (plan step) build work to Editor Agent.
The builders execute
Reproducer Agent (Generates reproduce.py to confirm the reported bug) and Editor Agent (Iteratively edits code, reusing AutoCodeRover search actions) build the work.
Independent review gates the work
Verifier Agent reviews the work — Runs reproduced + integration tests to confirm the fix. This reviewer is autonomous and separate from the agent that built the work, so the check is independent of its author.
Replicate this setup
Derived from the documented wiring — versions as declared by the owner.
Ingredients
Setup order
- 1.Stand up the orchestrator: Manager Agent.
- 2.Wire Editor Agent: it receives "dispatches (plan step)" from Manager Agent and sends "submits patch for test" to Verifier Agent. Wire Reproducer Agent: it receives "dispatches (plan step)" from Manager Agent and sends "reports reproduction" to Manager Agent. Wire Verifier Agent: it receives "submits patch for test" from Editor Agent and sends "reports verification result" to Manager Agent. Wire Fault Localizer Agent: it receives "dispatches (plan step)" from Manager Agent and sends "reports fault location" to Manager Agent.
Performance metrics
Windowed metrics with provenance. [unknown] means it was not tracked — an honest hole beats an invented figure.
85/300 resolved (one submission per issue). Source: arXiv 2406.01304. The paper's own Results prose separately states 'addressing 84 of 300' — an internal inconsistency in the paper itself (85 elsewhere, incl. the abstract and Table 4). [evidence_linked]
Token economics
Cost transparency is part of the honesty architecture. [unknown] means it was not tracked — not that it is zero.
≈299K tokens / $3.09 per issue avg; 30.39 API requests avg; hard cap $8/issue. Source: arXiv 2406.01304. [evidence_linked]
Blueprint
Operational DNA — why it works, how it was built, and how it is overseen. Not files for sale; knowledge of the design.
The ablation is the strongest evidence here: removing the multi-agent structure and task graph collapses performance from 22% to 10% on the same 50-issue subset — the paper's own data shows the coordination structure, not just the underlying model, drives most of the result.
Custom orchestration reusing search actions from SWE-agent and AutoCodeRover; plans encoded as JSON task graphs and parsed into an executable graph. Structured inter-agent reports only, no free-form chatter. Open-source at github.com/NL2Code/CodeR, but with no LICENSE file.
None in benchmark evaluation — the Manager autonomously selects a plan and coordinates the other four agents to a single submitted patch per issue.
Proof (1)
The team's shared track record — tasks, incidents, lessons, milestones. Per-entry provenance tags are always visible.
- ArtifactJun 1, 2024evidence-linked
CodeR paper published — arXiv 2406.01304
5-agent supervised pipeline: 28.33% (85/300) SWE-bench Lite, ~$3.09/issue avg. Ablation: full CodeR 22%, without multi-agent + task graph 10%, without fault localization 14% (50-issue subset). No LICENSE file exists in the repository.
https://arxiv.org/abs/2406.01304
Sign in to add a proof entry.
Sign inAttestations (0)
Named third-party statements from people with first-hand experience. Attestations are what separates Peer-Attested from Evidence-Linked.
No attestations yet. Worked with this configuration or agent? Attest to it using the form below — attestations are named third-party statements and are what separates Peer-Attested from Evidence-Linked.
Sign in to attest to this team.
Sign in