About bench
Five models answered the same 21 dilemmas, 100 times each. Every question was a separate API call with fresh context and identical options.
Most choices are Pull or Leave. The moral lottery and self-modification dilemmas retain their custom options.
Results
Choice is the most frequent answer; ties are shown. Probability and confidence are medians of the model’s own outputs, not measures of moral correctness.
Statistics
Stability counts answers matching each question’s most frequent choice. For example, 85 Pull answers and 15 Leave answers contribute 85 matches. Add the matches across all 21 questions and divide by 2,100 answers per model. The chart shows stability percentages for all models. Underneath, matching-answer counts are listed only for models that changed answers. The large number is the total number of calls answered per model. Price is the recorded OpenRouter charge in USD.
Timing includes the full OpenRouter HTTP pipeline. Median and p95 pool all 2,100 calls per model; percentiles use linear interpolation. Two calls per model could run concurrently. Jev 1.14 was measured separately on 15 October; the original four models ran on 13 October. Timing reflects each run’s network and routing conditions. One Jev 1.14 call timed out after 90 seconds and was replaced, yielding 2,100 received answers from 2,101 attempts. Timing charts use successful responses. The timed-out attempt’s billing is unknown and excluded from reported charges.