Opus 5.5
Claude Max
More tokens than usual 157/ 157
- Time
- 2m 30s
- Tokens
- 253K
- API price
- $0.56
- 5h limit
- 1%
Runs every morning at 6:00 ET
Daily model bench
Every morning each model I pay for gets the same kind of coding job: fix nine bugs, then build a scheduler from a written spec. The details change daily so nothing can be memorized, but the work stays the same size. If time, tokens or the pass rate move, the model or the plan changed.
last run Wednesday, Oct 28 · every run on GitHub
Fable 5.1
Claude Max
Slower than usual 157/ 157
- Time
- 4m 18s
- Tokens
- 216K
- API price
- $1.73
- 5h limit
- 3%
Sol 6.1
ChatGPT
Passed 157/ 157
- Time
- 2m 36s
- Tokens
- 162K
- API price
- n/a
- 5h limit
- n/a
Astra
ChatGPT
Passed 157/ 157
- Time
- 2m 28s
- Tokens
- 138K
- API price
- n/a
- 5h limit
- n/a
DeepSeek V4.1 Flash
OpenCode Go
More tokens than usual 157/ 157
- Time
- 12m 31s
- Tokens
- 2.8M
- API price
- $0.08
- 5h limit
- under 1%
Trends
Same work, every day
Each line is one model on the day's task. Hover for the numbers, or click a name to pick it out.
Pass record
One dot per day. Red means it missed at least one hidden test.
Opus 5.5 100%
Fable 5.1 100%
Sol 6.1 88%
Astra 100%
DeepSeek V4.1 Flash 100%
Today's twist
What changed in Oct 28's version of the task.
- Leftover cent goes to
- the earlier share
- Exact halves round
- to the even cent
- Fields look like
a=1| b='x'- Duration units
- h m s
- Day fields combine with
- AND, both must match
- Backwards ranges
- wrap around
- Search horizon
- 50 years
- Split function is called
allocate_units()
How it works
Fair, boring, every day
- Same work, new detailsNine bugs across five small modules, plus a cron-style scheduler built from a spec. A new variant is generated from the date, so every model gets the same task on the same day.
- Graded by hidden tests157 tests the model never sees check the work. Their answers come from a reference implementation that's itself checked against a brute-force search.
- No peekingEach model runs through its own CLI with fixed settings and none of my instructions. The hidden tests only exist after it's done, and any run that touches the grader is thrown out.
Today's models all pass this task nearly every time, so a red dot matters. Time, tokens and limit use are the trends to watch. Limit readings are whole percents shared with my other work, so a single day's number is rough.