SWE-bench
AssessTechniques
A benchmark for real-world software engineering bug-fix tasks.
Why it's here
Placed in Assess: 8 article(s) of evidence from 4 source(s), led by research-stage coverage, with 0 in the last 30 days. Confidence 71%.
Evidence (8)
- 8The New Stack·9/3/2026model_releaseIFM releases K2 Horizon open model fleet
IFM has released K2 Horizon, a connected fleet of six open models ranging from 0.9B to 375B parameters. The release includes weights, checkpoints, training code, configurations, logs, evaluation results, and data recipes under Apache 2.0, with the smaller models claiming state-of-the-art results in their size classes.
- 7The New Stack·8/26/2026framework_updateMicrosoft releases Agent Lightning v1.0 for agent RL
Microsoft has released Agent Lightning v1.0, a framework that keeps the production harness in control of context construction, tool execution, and agent-environment interactions during reinforcement learning. The company says it can improve coding-agent performance with modest compute, citing a Qwen3.5-9B result on SWE-bench Verified from 41.8% to 56.4% using 6K training examples.
- 7The New Stack·8/21/2026researchNew Benchmark Targets Large-Scale Code Refactoring
Researchers from several institutions introduced SWE-Bench ProMax, a multilingual benchmark focused on large-scale code refactoring across seven programming languages. The benchmark includes heavily curated real-world commit tasks and shows that current frontier AI coding agents still struggle, with the best model resolving only 41.2% of cases.
- 5GitHub Blog·7/22/2026product_launchGitHub explains what Copilot pricing covers versus raw API access
GitHub outlines the difference between paying for Copilot and using raw model APIs, arguing that Copilot includes the surrounding development workflow, policy controls, and billing visibility, not just model access. The post also says Copilot’s agentic harness can match task resolution with fewer tokens in many benchmark configurations, while raw APIs remain better for teams building their own systems.
- 7Hacker News·7/21/2026researchKimi K3 Matches Fable on Agent Tasks, Routing Together Improves Results
A benchmark study on about 1,030 real agentic tasks found Kimi K3 and Fable 5 performed at roughly similar quality overall, with each model showing strengths in different task categories. The post claims that routing between the two models achieved 93% accuracy and could cut costs by up to 50x versus using Fable alone on long agentic loops.
- 7Hacker News·7/21/2026model_releasePoolside releases Laguna S 2.1 coding model
Poolside has released Laguna S 2.1, a 118B-parameter Mixture-of-Experts coding model with 8B active parameters per token and support for up to a 1M-token context window. The company says it is optimized for longer-horizon agentic coding work and reports strong results on benchmarks such as Terminal-Bench 2.1, SWE-Bench Multilingual, SWE-Bench Pro, DeepSWE, SWE Atlas, and Toolathlon Verified.
- 7Hacker News·7/8/2026model_releaseCognition Launches SWE-1.7 Coding Model
Cognition announced SWE-1.7, its latest coding model, claiming it reaches frontier-level performance at lower cost and improves on prior results in agentic software engineering tasks. The company says the model was trained with RL pipeline improvements, better data quality, multi-cluster training, and self-compaction for long-horizon tasks, and is now available in Devin through Cerebras.
- 7OpenAI Blog·7/8/2026researchOpenAI Questions Reliability of SWE-Bench Pro
OpenAI published an analysis highlighting problems in SWE-Bench Pro, a widely used coding benchmark. The findings raise concerns that the benchmark may not reliably measure model performance or provide an accurate signal for evaluating AI coding systems.