All sessions

the makers this bench has never run

failed

Google, Meta, Mistral and MiniMax have no entry at all, which is four of the makers a reader would look for first. Current-generation and tool-capable only. key_limit is set per run from output price against the 64000-token ceiling: gemini pro reserves $0.77 of a key, maverick $0.05 / Started 31 Aug 2026, 16:31

Started
Completed
Planned15
Produced15
Missing0

Plan completion

Every requested trial remains visible, including runs that produced no record

SequenceReviewerRun identityLifecycleReasonInspect run
1afiafi/gemini-3.1-pro/20260831T163146Z completed-View evidence
2afiafi/gemini-3.1-pro/20260831T163147Z completed-View evidence
3afiafi/gemini-3.1-pro/20260831T163148Z completed-View evidence
4afiafi/gemini-3.7-flash/20260831T163149Z completed-View evidence
5afiafi/gemini-3.7-flash/20260831T163150Z completed-View evidence
6afiafi/gemini-3.7-flash/20260831T163151Z completed-View evidence
7afiafi/minimax-m3/20260831T163152Z completed-View evidence
8afiafi/minimax-m3/20260831T163153Z completed-View evidence
9afiafi/minimax-m3/20260831T163154Z completed-View evidence
10afiafi/devstral-2512/20260831T163155Z failed-View evidence
11afiafi/devstral-2512/20260831T163156Z failed-View evidence
12afiafi/devstral-2512/20260831T163157Z failed-View evidence
13afiafi/llama-4-maverick/20260831T163158Z failed-View evidence
14afiafi/llama-4-maverick/20260831T163159Z failed-View evidence
15afiafi/llama-4-maverick/20260831T163200Z failed-View evidence

Produced runs

View evidence opens one run detail

Details
afi
afi · cli
default Default profile · Meta · llama 4 maverick high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Meta · llama 4 maverick high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Meta · llama 4 maverick high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Mistral AI · devstral 2512 high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Mistral AI · devstral 2512 high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Mistral AI · devstral 2512 high
proxyN/AN/ANot recorded unscoredView evidence
afi
afi · cli
default Default profile · Minimax · m3 high
proxy15%N/A4 scoredView evidence
afi
afi · cli
default Default profile · Minimax · m3 high
proxy10%N/A2 scoredView evidence
afi
afi · cli
default Default profile · Minimax · m3 high
proxy20%N/A4 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.7 flash high
proxy15%N/A3 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.7 flash high
proxy35%N/A7 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.7 flash high
proxy25%N/A5 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.1 pro preview high
proxy45%N/A9 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.1 pro preview high
proxy20%N/A5 scoredView evidence
afi
afi · cli
default Default profile · Google · gemini 3.1 pro preview high
proxy40%N/A8 scoredView evidence

Session specification

The committed JSON is shown as data, not interpreted as instructions

{
  "defaults": {
    "effort": "high",
    "mode": "agent",
    "pr": "smykla-skalski/cairn#8",
    "subject": "proxy",
    "timeout": 900
  },
  "description": "Google, Meta, Mistral and MiniMax have no entry at all, which is four of the makers a reader would look for first. Current-generation and tool-capable only. key_limit is set per run from output price against the 64000-token ceiling: gemini pro reserves $0.77 of a key, maverick $0.05.",
  "name": "the makers this bench has never run",
  "runs": [
    {
      "key_limit": 8,
      "label": "gemini-3.1-pro",
      "model": "openrouter:google/gemini-3.1-pro-preview",
      "repeat": 3
    },
    {
      "key_limit": 4,
      "label": "gemini-3.7-flash",
      "model": "openrouter:google/gemini-3.7-flash",
      "repeat": 3
    },
    {
      "key_limit": 3,
      "label": "minimax-m3",
      "model": "openrouter:minimax/minimax-m3",
      "repeat": 3
    },
    {
      "key_limit": 3,
      "label": "devstral-2512",
      "model": "openrouter:mistralai/devstral-2512",
      "repeat": 3
    },
    {
      "key_limit": 2,
      "label": "llama-4-maverick",
      "model": "openrouter:meta-llama/llama-4-maverick",
      "repeat": 3
    }
  ]
}

Session evidence

Committed specification and session record

record present

Open verified source artifact

Open this path at the exported commit

The artifact is present; its content is available through the verified source link

spec present

Open verified source artifact

Open this path at the exported commit

The artifact is present; its content is available through the verified source link

benchee benchee-dashboard-1 built from 10f4ec58 Static benchmark evidence ·