We Benchmarked the DIY Alternative to Jev: Close on Price, Overconfident Until You Calibrate It

We did not test Jev. We tested the do-it-yourself alternative — a small LLM forced to answer in one token — over 3,400 calls against human labels. Against Jev's published numbers it is close on price, 7 to 21 points less accurate, and badly overconfident: 95% average confidence at 70% accuracy. One parameter, fitted on 200 labelled examples, fixed most of that.

TypeSafe launched Jev on 15 September 2026 with three headline claims: 40–200× faster, 444× cheaper, and incapable of hallucinating. The obvious developer objection is "I can fake that with logit_bias." This post measures the fake, and sets it beside what has been published about the real thing.

The verdict

1. Cheaper? Not by 444×. Against the cheapest hosted LLM, Jev is somewhere between the same price and about 2.5× cheaper, depending on how its tokens are counted. The 444× is against frontier models.
2. More accurate? On this task, its published number says yes — 90.5% against 70–83% for everything we ran. On other tasks, independent testing has it losing to Claude Haiku 4.5. There is no ranking to inherit.
3. The DIY confidence is a trap until you calibrate it. Raw, it is wildly overconfident. With one fitted parameter it lands in the same range as Jev's published calibration for choice questions. What Jev sells is getting there with no labelled data.

What Jev is, in two pictures

A normal LLM answers by writing text one token at a time. Jev skips that. You send a state — a ticket, a document, a trace — and typed questions, and it returns typed answers with probabilities in one forward pass.

There are three question types:

Type	Asks	Returns
noul	Is this true?	probability 0–1
choice	Which one of these? (≤255 options)	option, per-option probabilities, confidence
score	Where on a 2–10 level scale?	level, per-level probabilities, confidence

All questions are scored in parallel against the same state, so asking four costs about the same time as asking one. That parallel panel is the genuinely new idea.

The DIY alternative, and three things that bit us

Here is the imitation: bias the logits so only four tokens are legal, generate one token, read the confidence from the logprobs.

letters = {"A": "world", "B": "sports", "C": "business", "D": "scitech"}
bias = {str(enc.encode(l)[0]): 100 for l in letters}   # only these 4 are legal

resp = client.chat.completions.create(
    model="gpt-4.1-nano", logit_bias=bias, max_tokens=1, temperature=0,
    logprobs=True, top_logprobs=8,
    messages=[{"role": "system", "content": LETTER_PROMPT},
              {"role": "user", "content": article},
              {"role": "assistant", "content": "Answer:"}],
)
top = resp.choices[0].logprobs.content[0].top_logprobs

# Exact-match the four letters. Do NOT strip whitespace first: the API
# returns pre-bias logprobs, so " C" sits in the list next to "C".
probs = {t.token: math.exp(t.logprob) for t in top if t.token in letters}
route = letters[max(probs, key=probs.get)]
confidence = max(probs.values()) / sum(probs.values())

It is type-safe by construction and costs one output token. Running it taught us three things that matter more than the headline numbers:

• It does not work on current OpenAI models. Every GPT-5.x model we tried returns HTTP 400: 'logit_bias' is not supported with this model. You need an older hosted model (gpt-4.1-nano, gpt-4o-mini) or a self-hosted one — vLLM and llama.cpp support guided choice natively.
• The letter indirection can cost accuracy. On gpt-4.1-nano, answering with a letter scored 12 points below plain JSON mode on the same 200 cases (71.0% vs 83.0%, exact McNemar p < 0.0001). On gpt-4o-mini it made no difference (83.0% vs 82.0%).
• The logprobs are a minefield. They are reported before the bias is applied, and near-duplicate tokens (" C", "Ｃ", "\tC") appear beside the real one. Normalise after stripping whitespace and " C" (probability ≈ 0) silently overwrites "C" (probability ≈ 1) — a real bug we hit while building this, and it corrupts exactly the confidences you're trying to measure. Exact-match your tokens; do not trim first.

For comparison, the same decision in strict JSON mode gives you no confidence at all, and in Jev it is one call:

resp = client.post("https://api.typesafe.ai/v1/decisions", json={
    "model": "jev-latest",
    "state": article,
    "questions": {"topic": {
        "type": "choice",
        "instructions": "Classify this news article into one topic.",
        "criteria": {"world": "...", "sports": "...",
                     "business": "...", "scitech": "..."},
    }},
})
answer = resp.json()["answers"]["topic"]
topic, confidence = answer["choice"], answer["confidence"]

The three claims, against the numbers

Claim 1: "444× cheaper"

TypeSafe's comparison puts Jev against models like Claude Opus 5 and GPT-5.6. Nobody cost-conscious classifies tickets with Opus, so we priced the options a cost-conscious team would run, on AG News — a dataset where published Jev numbers exist.

Option	Accuracy	$ per 1M decisions	Source
jeff, self-hosted 400M GLiFormer	75.5%	~$2.60	third-party
Jev	90.5%	~$15.60 quoted · ~$6.70 at our token count	third-party · our arithmetic
gpt-4.1-nano, 1 constrained token	71.0%	$16.30	measured
gpt-4.1-nano, strict JSON	83.0%	$20.74	measured
gpt-4o-mini, 1 constrained token	83.0%	$24.45	measured
gpt-4o-mini, strict JSON	82.0%	$31.13	measured
gpt-5.6-luna, strict JSON	82.0%	$53.95	measured

The Jev arithmetic, since we did not measure it. Jev's list price is $0.042 per million input tokens with free output. The jeff project quotes ~$15.60 per million single-question requests, which works out to about 371 billed tokens per request. Our prompts averaged 159 input tokens; at list price that would be 159 × $0.042 ≈ $6.70 per million. Jev bills its own tokenisation of the state plus the question text, so we cannot say which figure is right for this task — only that the honest range is "parity to about 2.5× cheaper than gpt-4.1-nano," not 444×.

Two more things the table shows. Spending more on the LLM bought nothing: Luna, one of TypeSafe's own comparators, scored the same as gpt-4.1-nano in JSON mode (exact McNemar p = 0.83) at 2.6× the cost. And the only option that is clearly cheaper than Jev is self-hosting, at a real accuracy cost. One hidden cost on the Jev side, reported by independent testing: no prefix caching, so re-asking questions of the same large state pays for that state every time.

Claim 2: "40–200× faster"

We cannot verify or refute this, and we will not pretend to. Our calls ran 620–1,120 ms p50 measured from our client in India, network included. Published Jev figures (~130 ms on AG News from the jeff project; 0.4 s on TypeSafe's own workflows) are server-side. Those are not comparable measurements. TypeSafe itself describes its multipliers as "on the higher end of real world gains," measured from company laptops.

What we can say is arithmetic, not measurement: a decision is rarely the whole request.

If a frontier model was making the decision, replacing it roughly halves the request. If a small model already was, the end-to-end saving is a few percent, because generation dominates. Measure the decision's share of your latency before migrating it.

Claim 3: "Can't hallucinate"

TypeSafe is explicit that its 0% figure "is not empirical" and follows from schema matching. The guarantee is real: the answer is always one of your options. It is not a correctness claim, and the failure it leaves behind is the quieter one.

We also measured how much the guarantee buys you today. Strict JSON mode produced zero unparseable responses in 600 calls. On modern structured outputs, the failure Jev removes is already rare. What remains on every option here is the answer that is well-formed and wrong.

And accuracy: it depends on the task

On AG News, Jev's published 90.5% is above every cheap LLM we measured. An independent pre-registered test, grading against human labels on two production jobs, found the ranking flips:

Job	Jev	Claude Haiku 4.5
Commit-message classification (798 items)	65.8%	54.6%
Knowledge-base categorisation (450 items)	90.7%	97.8%

That study also tested abstention on the second job. Keeping each model's most confident 80%, both reached 98.6% — but on the same items Jev chose to keep, Haiku still led 98.9% to 98.6%. The author's conclusion is that abstention does not rescue Jev on that job, and he explicitly declines to call the result a calibration claim. The 98.6% figure alone reads as a bigger win than the source claims it is.

The part we got wrong, and what the data actually shows

Does the DIY classifier's confidence mean anything? We ran a design fixed before the calls were made: three prompt variants × two models, 200 test cases plus a separate 200-case holdout, confidence read only from the four allowed tokens.

Raw confidence is overconfident, not inverted.

Model · variant	Accuracy	Mean confidence	ECE (raw)	Right when ≥ 0.9	Right when < 0.8
nano · letters, fixed order	69.5%	95.2%	0.258	74.3% (n=171)	35.0% (n=20)
nano · letters, shuffled	69.5%	95.7%	0.267	73.6% (n=174)	40.0% (n=15)
nano · label words	71.0%	68.5%	0.251	80.9% (n=115)	57.7% (n=85)
4o-mini · letters, fixed order	82.5%	99.5%	0.175	82.7% (n=197)	50.0% (n=2)
4o-mini · letters, shuffled	82.0%	99.4%	0.176	82.7% (n=197)	33.3% (n=3)
4o-mini · label words	80.0%	97.6%	0.176	82.7% (n=191)	22.2% (n=9)

High-confidence answers are more accurate than low-confidence ones in every row (Fisher exact p ≤ 0.014 for all three nano variants). The problem is that almost everything is high-confidence: gpt-4o-mini reports ≥ 0.9 on 197 of 200 answers while getting 17% of them wrong. A threshold on that number filters nothing.

One class does most of the damage. gpt-4.1-nano with fixed letters, rows = true class:

	→ world	→ sports	→ business	→ sci/tech
world	47	3	0	0
sports	5	42	3	0
business	12	1	35	2
sci/tech	11	2	22	15

Sci/tech recall is 30%; most of it is filed under business, confidently. Shuffling which letter each class gets raised sci/tech recall to 48% without changing overall accuracy, so part of this is position or letter bias on the weakest model, and part is a genuinely blurry boundary (a story about a tech company's earnings is both). It is a property of this prompt and model, not of softmax confidence in general.

Temperature scaling fixes most of it. One parameter T, chosen to minimise log-loss on the 200 holdout cases, then applied to the 200 test cases:

Model · variant	T	ECE before	ECE after
nano · letters, fixed order	4.6	0.258	0.074
nano · letters, shuffled	5.5	0.267	0.055
nano · label words	6.2	0.251	0.166
4o-mini · letters, fixed order	7.8	0.175	0.055
4o-mini · letters, shuffled	7.3	0.176	0.045
4o-mini · label words	4.6	0.176	0.026

After scaling, the confidence is usable: for gpt-4o-mini with fixed letters, answers at ≥ 0.9 are right 94.7% of the time (n=75) and answers below 0.8 are right 55.6% (n=36). That is a gate you can act on.

For scale, independent testing reports Jev's calibration error at 0.086 for choice questions (n=2,600, three datasets), 0.012 for noul and 0.254 for score. Different data, different method — compare loosely. But the direction is clear: a calibrated DIY classifier lands in the same range as Jev's published choice calibration. It gets there with 200 labelled examples and one fitted number. Jev gets there with none.

What Jev is actually selling

Not cheaper classification — it is within about 2.5× of a small LLM. Not a guarantee of correctness. On the evidence available, it sells two things: higher zero-shot accuracy on some tasks, and calibrated confidence without any labelled data.

That is worth real money if you have dozens of decision types and no labels for any of them, or a contract that stops you collecting them. It is worth much less if you have one stable decision and a few hundred labelled examples, because then an afternoon of calibration gets you most of the way — on a model you may already be paying for, or one you host yourself.

What to do with this

1. Measure the share first. Time the decision against the whole request. If generation dominates, a faster decision layer will not move your latency or your bill.

2. Build a labelled holdout from your own traffic. Two hundred cases was enough to fit calibration and to detect a 12-point paired difference at p < 0.0001. Keep it separate from anything you tune on.

3. Never gate on raw confidence — from anything. Fit temperature scaling on the holdout, check ECE on separate data, and look at the confusion matrix before you trust an aggregate. A pipeline bug can produce a plausible-looking aggregate number that is still wrong; the confusion matrix is what catches it.

   def fit_temperature(holdout):            # holdout: [(logprobs[4], true_index)]
       def nll(T):
           return -sum(math.log(softmax([lp / T for lp in lps])[y])
                       for lps, y in holdout) / len(holdout)
       return min((0.25 * 1.06 ** i for i in range(90)), key=nll)
   

4. Pick where it runs.

If…	Use
Many decision types, no labelled data, hosted US inference is fine	Jev — and verify its confidence on your data, especially for score questions
You have a few hundred labels and one stable decision	A small LLM or fine-tuned encoder, calibrated
Data residency or on-prem is required	Self-hosted (jeff / GLiFormer, a fine-tuned encoder, or guided decoding on an open model) — calibrated the same way
Decisions are rare and the answer needs reasoning or prose	Keep the LLM

One budget note, from the name. TypeSafe named the model after William Stanley Jevons, whose paradox is that cheaper resources get used more, not less. At frontier-model prices you would never evaluate every frame or keystroke; at around $16 per million decisions, you will. Track cost and volume per decision point before the price drop multiplies them.

The one-line version: the cheap imitation is close on price and fixable on confidence — if you have labels. Jev is what you buy when you don't.

---

Appendix: method, data and limitations

Run 1 — cost, latency, accuracy (2026-09-21). 200 articles from the AG News test split, 50 per class, seed 20260921. Five configurations of three OpenAI models, one call per case, 1,000 calls, zero errors. gpt-5.6-luna was called with default settings (no reasoning-effort parameter); its output ran 11–82 tokens per call, mean 14, for a ~5-token JSON answer, and that is included in its cost. Prices: OpenAI Standard tier, verified the same day on OpenAI's pricing page. Latency is client-side from India, network included.

Configuration	Accuracy (95% Wilson CI)	p50	p95	$ per 1M
gpt-4.1-nano, 1 token	71.0% (64.4–76.8)	621 ms	893 ms	$16.30
gpt-4.1-nano, JSON	83.0% (77.2–87.6)	621 ms	996 ms	$20.74
gpt-4o-mini, 1 token	83.0% (77.2–87.6)	675 ms	870 ms	$24.45
gpt-4o-mini, JSON	82.0% (76.1–86.7)	704 ms	1,108 ms	$31.13
gpt-5.6-luna, JSON	82.0% (76.1–86.7)	1,124 ms	1,579 ms	$53.95

Run 2 — calibration (2026-09-21). Same 200 test cases plus a disjoint 200-case holdout (seed 20260922), three prompt variants × two models, 2,400 calls. The design — variants, holdout, metrics, the Fisher test — is written in the script's docstring and was fixed before the calls were made. ECE uses ten equal-width bins. gpt-4.1-nano with fixed letters scored 71.0% in run 1 and 69.5% in run 2 on identical inputs at temperature 0: the API is not fully deterministic, and differences of a point or two between runs are noise.

Everything is downloadable. Results: benchmark summary · benchmark calls · calibration summary · calibration calls with all four logprobs. Inputs: 200 test cases · 200 holdout cases · label descriptions. Code, including every prompt verbatim: bench-decision-layer.py · calibrate-decision-layer.py.

What is third-party. All Jev figures: TypeSafe's launch post and dashboard, the jeff project's AG News comparison, and an independent pre-registered test (~9,750 calls) for calibration by question type, the rank reversal and the abstention result. We reproduced none of them and had no Jev API access.

Limitations. One dataset, one task, four classes; news-topic classification is easier and cleaner than most production decisions. Two hundred cases gives accuracy intervals of about ±6 points and makes small bins noisy (several low-confidence bins above have n < 10). Temperature scaling was tested on data from the same distribution as the holdout; it will not survive distribution shift without refitting. Our Jev cost comparison rests on third-party numbers from a different AG News sample. Jev is a week old and in early access; TypeSafe expects prices to fall. This analysis was written up the same day the calls were made.

Written by Vikas Chamarthi, founder of NavyaAI, which does LLM inference cost and architecture audits. Benchmark code, data and analysis are ours; prose was drafted with AI assistance and reviewed by the named author. We have no commercial relationship with TypeSafe.

Sources: TypeSafe launch post · TypeSafe docs · Independent pre-registered test, Primeline · jeff, self-hosted System One · Hacker News discussion · Sean Goedecke on structured output · OpenAI API pricing