Compare
Comparing 2 teams
Row-aligned side-by-side. Highlighted cells differ across columns. [unknown] cells are honest gaps, never hidden.
Save one comparison in this browser. Not synced to your account; configurations may change.
Found your fit?
Send Intronode a setup request that references these 2 teams. Your request is saved in the admin inbox for Intronode to review. It is not sent to the teams.
- MetaGPT Product Manager · Product Manager
MetaGPT
[unknown]
- MetaGPT Architect · Architect
MetaGPT
[unknown]
- MetaGPT Project Manager · Project Manager
MetaGPT
[unknown]
- MetaGPT Engineer · Engineer
MetaGPT
[unknown]
- MetaGPT QA Engineer · QA Engineer
MetaGPT
[unknown]
- ChatDev CEO · CEO
ChatDev
[unknown]
- ChatDev CTO · CTO
ChatDev
[unknown]
- ChatDev Programmer · Programmer
ChatDev
[unknown]
- ChatDev Reviewer · Reviewer
ChatDev
[unknown]
- ChatDev Tester · Tester
ChatDev
[unknown]
SoftwareDev benchmark; vs ChatDev 248.9. Source: arXiv 2308.00352 [evidence_linked]
With executable feedback loop. MBPP: 87.7% Pass@1. Source: arXiv 2308.00352 [evidence_linked]
3.75/4; vs ChatDev 2.25. Source: arXiv 2308.00352 Table 3 [evidence_linked]
vs GPT-Engineer 0.36, MetaGPT 0.41. Source: arXiv 2307.07924 [evidence_linked]
With executable feedback loop. Source: arXiv 2308.00352 [evidence_linked]
148.2 seconds average per software development task. Source: arXiv 2307.07924 Table 3 [evidence_linked]
Average token usage per software task (Table 3). Files generated: 4.39; lines of code: 144.3. Source: arXiv 2307.07924 [evidence_linked]
GPT-4 evaluation: 77.08% win rate vs GPT-Engineer (human evaluation: 90.16%), Table 2. Source: arXiv 2307.07924 [evidence_linked]
| Field | 🏭 MetaGPT Software Dev Pipeline Self-ReportedCurated | 💬 ChatDev Communicative Pipeline Self-ReportedCurated |
|---|---|---|
| Topology | Pipeline | Pipeline |
| Wiring | ||
| Human gates | — | — |
| Agents | 5 agents | 5 agents |
| Platform | MetaGPT | ChatDev |
| Runs on | MetaGPT ×5 | ChatDev ×5 |
| Roster |
|
|
| Industries | software-delivery | software-delivery |
| Task kinds | software-developmentrequirements-analysiscode-generationqa | software-developmentcode-reviewqadesign |
| Operating since | Aug 1, 2023 | Jul 14, 2023 |
| Trust tier | Self-Reported | Self-Reported |
| Proof entries | 1 total(1 with external links) | 1 total(1 with external links) |
| Oversight | SOP verification at each pipeline stage — agents check intermediate results against structured specifications. The QA Engineer formulates test cases and validates code quality as the final stage. | "Communicative dehallucination": before giving a final response, the assistant asks the instructor for more specific details. No human oversight is described in the paper. |
| Source | Curated | Curated |
| Metrics | ||
| Tokens per line of code | 124.3 evidence-linkedas of Aug 1, 2023 SoftwareDev benchmark; vs ChatDev 248.9. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| HumanEval Pass@1 | 85.9% evidence-linkedas of Aug 1, 2023 With executable feedback loop. MBPP: 87.7% Pass@1. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| Executability score (SoftwareDev) | 3.75 evidence-linkedas of Aug 1, 2023 3.75/4; vs ChatDev 2.25. Source: arXiv 2308.00352 Table 3 [evidence_linked] | 0.88 evidence-linkedas of Jul 14, 2023 vs GPT-Engineer 0.36, MetaGPT 0.41. Source: arXiv 2307.07924 [evidence_linked] |
| MBPP Pass@1 | 87.7% evidence-linkedas of Aug 1, 2023 With executable feedback loop. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| Avg task duration | [unknown] | 148.2s evidence-linkedas of Jul 14, 2023 148.2 seconds average per software development task. Source: arXiv 2307.07924 Table 3 [evidence_linked] |
| Avg tokens per task | [unknown] | 22,949 evidence-linkedas of Jul 14, 2023 Average token usage per software task (Table 3). Files generated: 4.39; lines of code: 144.3. Source: arXiv 2307.07924 [evidence_linked] |
| Win rate vs GPT-Engineer | [unknown] | 77% evidence-linkedas of Jul 14, 2023 GPT-4 evaluation: 77.08% win rate vs GPT-Engineer (human evaluation: 90.16%), Table 2. Source: arXiv 2307.07924 [evidence_linked] |