JevBench Scores Models That Answer With a Choice, Not a Paragraph
A public benchmark for models that return bounded choices and probabilities rather than text: 534 English decisions weighted into one score across chance-corrected accuracy, calibration, speed and cost, with scoring code and per-task results released.
A model class that doesn’t write paragraphs
JevBench isn’t measuring prose. The models it targets — the author’s term is Jev-class — answer by picking from a bounded set of choices and attaching a probability. The input is still text. The author’s claim is that on that text these models are about as intelligent as an LLM while being substantially faster and cheaper. That claim is the reason the benchmark exists: it’s what lets someone else check the claim instead of taking it. The submission doesn’t explain where the speed and cost advantage comes from; the output format is the obvious candidate, since a label and a probability cost far fewer tokens to produce than a paragraph.
What the harness does is straightforward. A full run asks 534 English decision questions and reports accuracy, latency and price together rather than in separate tables. The three are combined into one weighted score, and the weights are configurable, so the same raw results can be ordered differently depending on whether the reader cares more about cost or about speed. That flexibility is useful and also a warning: a ranking is only as meaningful as the weighting behind it, and the weighting is a judgment call, not a measurement.
What the v1.3 score contains
Four parts: chance-corrected intelligence, calibration, speed, and cost. Chance correction is worth naming, because with a small answer space guessing earns real points; without it, accuracy looks better than it is. Calibration is the other sensible inclusion — probability output is only useful downstream if a stated 70% lands near 70% in practice, and a model that is confident and wrong is worse than one that hedges. Speed and cost are measured rather than estimated, subject to the latency caveat below.
The current top five, and what they don’t settle
- #1 Jev — 74.4
- #2 SemIf — 73.1
- #3 djev — 73.0
- #4 Winnow-12B Q8 — 71.2
- #5 reflex 4B — 70.3
First to second is 1.3 points. Second to third is 0.1. Fourth to fifth is 0.9.
Two things belong on the table before those numbers get quoted. First, the benchmark comes from someone who is openly on Jev’s side: the submission opens with “Jev kicks ass,” and the stated motivation is showing how the serious open source projects and the fake lookalikes actually compare. Jev takes the top slot. That isn’t evidence of anything improper, but who is grading whom is part of the context of a ranking. Second, the author lists the limitations: English only; latency from a single German server; a disclosed ×2 adjustment plus 150 ms applied to local and demo timings, described as an informed assumption rather than a measurement; held-out prompts still reaching the services under evaluation; and roughly one-point gaps that can be noise. By that last standard, the 0.1-point gap near the top of the list isn’t a result, and the 0.9 points between fourth and fifth sit right at the edge.
Where to check it yourself
The MIT-licensed harness, the public items, frozen artifacts, scoring code and per-task outcomes are at github.com/fstandhartinger/jevbench, along with two demos that don’t require signup. For a benchmark published by the same camp as its leading entry, that’s roughly the right disclosure: it doesn’t end the argument, but it makes the argument testable. Anyone who disagrees can install it and rerun the questions.
What would move these numbers from one camp’s result to a general finding is fairly clear: runs by people with no stake in Jev, questions in languages other than English, and latency measured somewhere other than a single German server. None of that is a criticism of the current release. It’s the distance between a benchmark and a verdict. The Hacker News thread is at 97 points and 24 comments.