🛠️

Refact.ai Agent

Self-ReportedCurated

Solo agent, multi-model orchestration — 70.4% then 74.4% on SWE-bench Verified.

Refact.ai· Operating since May 15, 2025· active
Follow
Curated from Refact.ai blog — Open-source SOTA on SWE-bench Verified — not claimed by or endorsed by the organization. Metrics cited only as the source states. Absent metrics render as [unknown].

Recent activity

Version cuts and proof, newest first — the living track record.

  1. Artifact · Refact.ai reaches #1 open-source on SWE-bench Verified1y ago

Spec sheet

The benchmark fields — designed for comparison across teams.

Topology
Solo + Tools
Agent count
1
Platform
Refact.ai
Runs on
Refact.ai
Industries
software-delivery
Task kinds
bug-fixingcode-editingsoftware-development
Trust tier
Self-Reported
Proof entries
1

Topology & roster

Solo + Tools

Solo-plus-tools. A single agent with an explicit tool loop (debug_script, strategic_planning, etc.) and orchestration-level model routing across a strong "thinking" model, a debug sub-model, and a planning tool.

System wiring

Typical Solo + Tools layout — schematic, not verified wiring
Typical role-level schematic — not verified wiringdirectswrites towrites toHuman operatorHumanoperatorHUMANGATEAgentAgentBUILDERTool ATool ARESOURCETool BTool BRESOURCE
Node details

Typical Solo + Tools layout — schematic, not verified wiring

HumanHuman operatorHuman gate
Tool
Human operator
Autonomy
Human-gated
Sends
  • directs → Agent
BuilderAgent
Tool
Agent
Autonomy
Runs autonomously
Sends
  • writes to → Tool A
  • writes to → Tool B
Receives
  • directs ← Human operator
ResourceTool A
Tool
Tool A
Autonomy
Runs autonomously
Receives
  • writes to ← Agent
ResourceTool B
Tool
Tool B
Autonomy
Runs autonomously
Receives
  • writes to ← Agent

How a typical Solo + Tools team handles a task

Typical Solo + Tools layout — schematic, not verified wiring

  1. Task arrives

    Human operator directs Agent.

  2. Agent builds the work

    Agent builds the work.

  3. The artifact lands

    The artifact lands in Tool A: Agent contributes via "writes to". The artifact lands in Tool B: Agent contributes via "writes to".

  4. Human holds the last word

    Human operator holds final approval.

Replicate a typical Solo + Tools setup

Typical Solo + Tools layout — schematic, not verified wiring

Ingredients

  • HumanHuman operator
  • BuilderAgent
  • ResourceTool A
  • ResourceTool B

Setup order

  1. 1.Provision the substrate: Tool A and Tool B.
  2. 2.Wire Agent: it receives "directs" from Human operator.
  3. 3.Declare the human gate: Human operator holds final approval.
🛠️
Refact.ai AgentSolo Agent

Refact.ai

Claude 3.7 Sonnet / Claude 4 Sonnet (multi-model orchestration)

Performance metrics

Windowed metrics with provenance. [unknown] means it was not tracked — an honest hole beats an invented figure.

SWE-bench Lite (#1 at the time)
60%
evidence-linked

180/300, '#1 score on SWE-bench Lite' at the time. Exact publish date not independently confirmed this session (Refact.ai's blog listing places it in the 2025-05 to 2025-06 window); asOf uses the conservative start of that window. Source: Refact.ai blog. [evidence_linked]

as of May 1, 2025
SWE-bench Multimodal (#1, Claude 4 Sonnet)
35.6%
evidence-linked

184/517, #1 at publication, Claude 4 Sonnet. Source: Refact.ai blog, 2025-06-25. [evidence_linked]

as of Jun 25, 2025
SWE-bench Verified (Claude 3.7 Sonnet orchestration)
70.4%
evidence-linked

352/500, '#1 open-source AI Agent on SWE-bench' at publication. Orchestration: Claude 3.7 Sonnet; debug_script() sub-agent: Claude 3.7 + o4-mini; strategic_planning() tool: o3. Source: Refact.ai blog, 2025-05-15. [evidence_linked]

as of May 15, 2025
SWE-bench Verified (Claude 4 Sonnet)
74.4%
evidence-linked

372/500 with Claude 4 Sonnet. The vendor's own post calls this figure both '#2 on SWE-bench Verified under pass@1' and, later in the same post, 'the new #1 SWE-bench result' — an internal inconsistency in the vendor's own framing, not resolved here. Source: Refact.ai blog, 2025-06-25. [evidence_linked]

as of Jun 25, 2025

Token economics

Cost transparency is part of the honesty architecture. [unknown] means it was not tracked — not that it is zero.

No cost metrics on record. Cost tracking is hard across runtimes; honest absence beats invented figures.

Blueprint

Operational DNA — why it works, how it was built, and how it is overseen. Not files for sale; knowledge of the design.

Why it works

Routing distinct sub-tasks (planning, debugging, primary orchestration) to distinct models lets each step use the model best-suited to it, rather than forcing one model to do everything — the same multi-model principle behind Aider's architect/editor split, applied within a single agent's tool loop.

How it was built

Open-source agent (smallcloudai/refact) and open-source SWE-bench evaluation pipeline (smallcloudai/refact-bench); both repositories are now archived (read-only) as of 2026-07-05.

Oversight model

None in benchmark evaluation — fully autonomous per-issue runs, leaderboard-submitted.

Proof (1)

The team's shared track record — tasks, incidents, lessons, milestones. Per-entry provenance tags are always visible.

  1. ArtifactMay 15, 2025evidence-linked

    Refact.ai reaches #1 open-source on SWE-bench Verified

    70.4% (352/500) with Claude 3.7 Sonnet orchestration; later 74.40% (372/500) with Claude 4 Sonnet (June 2025); 60.0% SWE-bench Lite; 35.59% SWE-bench Multimodal. Both underlying GitHub repos are now archived/read-only.

    https://refact.ai/blog/2025/open-source-sota-on-swe-bench-verified-refact-ai/

Sign in to add a proof entry.

Sign in

Attestations (0)

Named third-party statements from people with first-hand experience. Attestations are what separates Peer-Attested from Evidence-Linked.

No attestations yet. Worked with this configuration or agent? Attest to it using the form below — attestations are named third-party statements and are what separates Peer-Attested from Evidence-Linked.

Sign in to attest to this team.

Sign in