Terminal-bench-2-1 Alternatives
Similar projects and alternatives to terminal-bench-2-1
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
terminal-bench-2-1 discussion
terminal-bench-2-1 reviews and mentions
-
Introducing Muse Spark 1.1
Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness given a situation.
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, usually weaker models often succeed.
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the buggy versions. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure (from what I have seen) is the task timeout hitting before the agent finishes
[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...
[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...
- OpenAI no longer recommends SWE-Bench Pro
Stats
harbor-framework/terminal-bench-2-1 is an open source project licensed under Apache License 2.0 which is an OSI approved license.
The primary programming language of terminal-bench-2-1 is Shell.
Sponsored