We Benchmarked Jev Directly: More Accurate, Not Cheaper, and Calibration Isn't the Moat

TypeSafe gave us API access, so we put Jev to the test: 200 real API calls, each one checked against human labels, using the same setup we'd used for the DIY and self-hosted alternatives. Jev is the clear accuracy winner, 87.5–88% compared to 70–83% for cheaper LLMs and 75% for jeff. It costs more, though: $18.57 per million decisions versus $16.30 if you build it yourself. Its calibration is good, but a free self-hosted model already matches it straight out of the box, no tuning needed.

TypeSafe launched Jev on September 15, 2026, making some bold promises: 40–200× faster, 444× cheaper, and impossible to hallucinate. The natural response from developers is, "I can just fake that with logit_bias." So, we put that to the test, tried jeff's free self-hosted version, and now, with actual API access, we've finally benchmarked Jev itself on those same 200 cases.

What we found

1. Is Jev actually cheaper? Not in our experience. We measured it ourselves: Jev costs $18.57 per million decisions, which is more than the cheapest DIY setup we tested ($16.30, using one constrained token on gpt-4.1-nano). That "444× cheaper" claim only holds if you compare against frontier models that nobody trying to save money would actually use for this job.
2. Jev really is more accurate. 87.5–88% compared to 70–83% for the LLMs we tested, and 75% for self-hosted jeff. That's the main advantage. But don't expect these numbers to generalize: independent tests on other tasks have Jev losing to Claude Haiku 4.5, so don't assume the ranking holds everywhere.
3. You need to calibrate both, not just the DIY route. Out of the box, DIY models are wildly overconfident (ECE as high as 0.267). Jev's raw confidence is a bit better at 0.080, and with temperature scaling, it drops to 0.037. Still, that isn't clearly better than a free self-hosted classifier, which comes in at 0.053 with zero tuning.

What Jev is, in two pictures

A normal LLM answers by writing text one token at a time. Jev skips that. You send a state (a ticket, a document, a trace) and typed questions, and it returns typed answers with probabilities in one forward pass.

There are three question types:

Type	Asks	Returns
noul	Is this true?	probability 0–1
choice	Which one of these? (≤255 options)	option, per-option probabilities, confidence
score	Where on a 2–10 level scale?	level, per-level probabilities, confidence

All questions are scored in parallel against the same state, so asking four costs about the same time as asking one. That parallel panel is the genuinely new idea.

The DIY alternative, and three things that bit us

Here is the imitation: bias the logits so only four tokens are legal, generate one token, read the confidence from the logprobs.

letters = {"A": "world", "B": "sports", "C": "business", "D": "scitech"}
bias = {str(enc.encode(l)[0]): 100 for l in letters}   # only these 4 are legal

resp = client.chat.completions.create(
    model="gpt-4.1-nano", logit_bias=bias, max_tokens=1, temperature=0,
    logprobs=True, top_logprobs=8,
    messages=[{"role": "system", "content": LETTER_PROMPT},
              {"role": "user", "content": article},
              {"role": "assistant", "content": "Answer:"}],
)
top = resp.choices[0].logprobs.content[0].top_logprobs

# Exact-match the four letters. Do NOT strip whitespace first: the API
# returns pre-bias logprobs, so " C" sits in the list next to "C".
probs = {t.token: math.exp(t.logprob) for t in top if t.token in letters}
route = letters[max(probs, key=probs.get)]
confidence = max(probs.values()) / sum(probs.values())

It is type-safe by construction and costs one output token. Running it taught us three things that matter more than the headline numbers:

• It does not work on current OpenAI models. Every GPT-5.x model we tried returns HTTP 400: 'logit_bias' is not supported with this model. You need an older hosted model (gpt-4.1-nano, gpt-4o-mini) or a self-hosted one; vLLM and llama.cpp support guided choice natively.
• The letter indirection can cost accuracy. On gpt-4.1-nano, answering with a letter scored 12 points below plain JSON mode on the same 200 cases (71.0% vs 83.0%, exact McNemar p < 0.0001). On gpt-4o-mini it made no difference (83.0% vs 82.0%).
• The logprobs are a minefield. They are reported before the bias is applied, and near-duplicate tokens (" C", "Ｃ", "\tC") appear beside the real one. Normalise after stripping whitespace and " C" (probability ≈ 0) silently overwrites "C" (probability ≈ 1): a real bug we hit while building this, and it corrupts exactly the confidences you're trying to measure. Exact-match your tokens; do not trim first.

For comparison, the same decision in strict JSON mode gives you no confidence at all, and in Jev it is one call. This is the exact request we used for the measurements below:

resp = client.post("https://api.typesafe.ai/v1/systemone", json={
    "model": "jev-latest",
    "state": article,
    "questions": {"topic": {
        "type": "choice",
        "instructions": "Classify this news article into one topic.",
        "criteria": {"world": "...", "sports": "...",
                     "business": "...", "scitech": "..."},
    }},
})
answer = resp.json()["answers"]["topic"]
topic, confidence = answer["choice"], answer["confidence"]

The three claims, against the numbers

1. "444× cheaper"

TypeSafe's comparison puts Jev against models like Claude Opus 5 and GPT-5.6. Nobody cost-conscious classifies tickets with Opus, so we priced the options a cost-conscious team would run, on AG News, now including Jev itself, measured on the same 200 cases as everything else in this table.

Option	Accuracy	$ per 1M decisions	Source
jeff, self-hosted 400M GLiFormer	75.5%	~$2.60	third-party
gpt-4.1-nano, 1 constrained token	71.0%	$16.30	measured
Jev	87.5–88.0%	$18.57	measured, first-party
gpt-4.1-nano, strict JSON	83.0%	$20.74	measured
gpt-4o-mini, 1 constrained token	83.0%	$24.45	measured
gpt-4o-mini, strict JSON	82.0%	$31.13	measured
gpt-5.6-luna, strict JSON	82.0%	$53.95	measured

200 real calls to the production API averaged 442 input tokens and 47.3 output tokens per decision: a full news article plus four criteria descriptions, batched into one choice question. At list price ($0.042 per million input tokens) that comes to $18.57 per million decisions, above the cheapest DIY option and well above the $6.70 estimate we published before we had API access. A real four-class prompt with concrete criteria runs much bigger than the single-topic-string assumption behind that earlier arithmetic. Jev is not cheaper than the cheapest DIY route on this task, and it isn't "parity to 2.5× cheaper" either; that estimate was simply wrong.

Two more things the table shows. Spending more on the LLM bought nothing: Luna, one of TypeSafe's own comparators, scored the same as gpt-4.1-nano in JSON mode (exact McNemar p = 0.83) at 2.6× the cost. And the only option clearly cheaper than Jev is self-hosting, 12 points less accurate for about a seventh of the price. Independent testing also flagged a hidden Jev cost: no prefix caching, so re-asking questions about the same large state pays for that state every time. Our own token count reflects a fresh state on every call, so it doesn't include that penalty.

2. "40–200× faster"

We still can't verify the published 40–200× multiplier: it's measured against frontier models we didn't call in this comparison, and TypeSafe itself describes it as "on the higher end of real-world gains," clocked from company laptops. What we do have now is real Jev latency, from the same client, against the same DIY options, on the same network path. Jev's calls ran 357.7 ms p50 and 454.9 ms p95 from India, network included, against 621 ms for gpt-4.1-nano on one constrained token and 675 ms for gpt-4o-mini. Call it a real 1.7–1.9× speedup over the DIY route. Nowhere near 40–200×, but not nothing.

What we can say is arithmetic, not measurement: a decision is rarely the whole request.

If a frontier model was making the decision, replacing it roughly halves the request. If a small model already was, the end-to-end saving is a few percent, because generation dominates. Measure the decision's share of your latency before migrating it.

3. "Can't hallucinate"

TypeSafe is explicit that its 0% figure "is not empirical" and follows from schema matching. The guarantee is real: the answer is always one of your options. It is not a correctness claim, and the failure it leaves behind is the quieter one.

We also measured how much the guarantee buys you today. Strict JSON mode produced zero unparseable responses in 600 calls. On modern structured outputs, the failure Jev removes is already rare. What remains in every option is an answer that is well-formed and wrong.

Accuracy depends on the task

On AG News, Jev's published 90.5% is above every cheap LLM we measured. An independent pre-registered test, grading against human labels on two production jobs, found the ranking flips:

Job	Jev	Claude Haiku 4.5
Commit-message classification (798 items)	65.8%	54.6%
Knowledge-base categorisation (450 items)	90.7%	97.8%

That study also tested abstention on the second job. Keeping each model's most confident 80%, both reached 98.6%, but on the same items Jev chose to keep, Haiku still led 98.9% to 98.6%. The author concludes that abstention does not rescue Jev on that job, and he explicitly declines to call the result a calibration claim. The 98.6% figure alone reads as a bigger win than the source claims.

The part we got wrong, and what the data actually shows

Does the DIY classifier's confidence mean anything? We ran a design fixed before the calls were made: three prompt variants × two models, 200 test cases plus a separate 200-case holdout, confidence read only from the four allowed tokens.

Raw confidence is overconfident, not inverted.

Model · variant	Accuracy	Mean confidence	ECE (raw)	Right when ≥ 0.9	Right when < 0.8
nano · letters, fixed order	69.5%	95.2%	0.258	74.3% (n=171)	35.0% (n=20)
nano · letters, shuffled	69.5%	95.7%	0.267	73.6% (n=174)	40.0% (n=15)
nano · label words	71.0%	68.5%	0.251	80.9% (n=115)	57.7% (n=85)
4o-mini · letters, fixed order	82.5%	99.5%	0.175	82.7% (n=197)	50.0% (n=2)
4o-mini · letters, shuffled	82.0%	99.4%	0.176	82.7% (n=197)	33.3% (n=3)
4o-mini · label words	80.0%	97.6%	0.176	82.7% (n=191)	22.2% (n=9)

High-confidence answers are more accurate than low-confidence ones in every row (Fisher exact p ≤ 0.014 for all three nano variants). The problem is that almost everything is high-confidence: gpt-4o-mini reports ≥ 0.9 on 197 of 200 answers while getting 17% of them wrong. A threshold on that number filters nothing.

One class does most of the damage. gpt-4.1-nano with fixed letters, rows = true class:

	→ world	→ sports	→ business	→ sci/tech
world	47	3	0	0
sports	5	42	3	0
business	12	1	35	2
sci/tech	11	2	22	15

Sci/tech recall is 30%; most of it is filed under business, confidently. Shuffling which letter each class gets raised sci/tech recall to 48% without changing overall accuracy, so part of this is position or letter bias on the weakest model, and part is a genuinely blurry boundary (a story about a tech company's earnings is both). It is a property of this prompt and model, not of softmax confidence in general.

Temperature scaling fixes most of it. One parameter T, chosen to minimise log-loss on the 200 holdout cases, then applied to the 200 test cases:

Model · variant	T	ECE before	ECE after
nano · letters, fixed order	4.6	0.258	0.074
nano · letters, shuffled	5.5	0.267	0.055
nano · label words	6.2	0.251	0.166
4o-mini · letters, fixed order	7.8	0.175	0.055
4o-mini · letters, shuffled	7.3	0.176	0.045
4o-mini · label words	4.6	0.176	0.026

After scaling, the confidence is usable: for gpt-4o-mini with fixed letters, answers at ≥ 0.9 are right 94.7% of the time (n=75) and answers below 0.8 are right 55.6% (n=36). That is a gate you can act on.

Here's the direct comparison, same method, same 200 test cases and the same disjoint holdout:

Backend	Accuracy	ECE (raw)	ECE (temp-scaled)	Right when ≥ 0.9	Right when < 0.8
Jev	88.0%	0.080	0.037 (T=5.17)	92.1% (n=176)	52.9% (n=17)
jeff, self-hosted	75.0%	0.053	0.056	n/a	n/a
gpt-4o-mini · label words, temp-scaled	80.0%	0.176	0.026	n/a	n/a
gpt-4.1-nano · letters, fixed order, temp-scaled	69.5%	0.258	0.074	74.3% (n=171)	35.0% (n=20)

Jev's raw confidence is meaningfully better calibrated than any raw DIY output, so that part of the pitch holds. Two things complicate "calibration without labels" as the reason to pay for it, though. jeff's raw ECE of 0.053, with no fitting at all, already beats Jev's raw 0.080; self-hosting the free option gets you closer to calibrated out of the box. And once you have a 200-case holdout, which the DIY route needs anyway and which sharpens Jev too, temperature scaling pulls the best configurations into the same range: 0.026 for the best DIY setup, 0.037 for Jev. Fitting temperature scaling on Jev's own output still helps, 0.080 down to 0.037, so "no labelled data needed" holds for the zero-shot number. But the calibration edge over a free self-hosted model is smaller than the published numbers suggest. In the raw, no-labels case, jeff is ahead.

For scale, independent testing reports Jev's calibration error at 0.086 for choice questions (n=2,600, three datasets), close to our own 0.080 though from different data and method, so compare loosely. Its figures for the other two types: 0.012 for noul, 0.254 for score.

What Jev is actually selling

It isn't cheaper classification; we measured it above the cheapest DIY option, not below. It isn't a correctness guarantee, and its calibration edge over a free self-hosted model doesn't hold up in raw form. What the evidence, now first-party, actually supports is simpler: higher zero-shot accuracy with no labelled data required: 87.5–88% against 70–83% for the LLM configs we ran and 75% for self-hosted jeff, on this task.

That's worth real money if a decision needs to be right out of the box: dozens of decision types, no labels for any of them, or a contract that stops you from collecting them. It's worth much less with one stable decision and a few hundred labelled examples, since an afternoon of calibration gets a free self-hosted model most of the way to Jev's calibration, leaving the accuracy gap as the only thing left to weigh against $18.57 per million decisions.

What to do with this

1. Measure the share first. Time the decision against the whole request. If generation dominates, a faster decision layer will not move your latency or your bill.

2. Build a labelled holdout from your own traffic. Two hundred cases were enough to fit calibration and to detect a 12-point paired difference at p < 0.0001. Keep it separate from anything you tune on.

3. Never gate on raw confidence, from anything. Fit temperature scaling on the holdout, check ECE on separate data, and look at the confusion matrix before you trust an aggregate. A pipeline bug can produce a plausible-looking aggregate number that is still wrong; the confusion matrix is what catches it.

   def fit_temperature(holdout):            # holdout: [(logprobs[4], true_index)]
       def nll(T):
           return -sum(math.log(softmax([lp / T for lp in lps])[y])
                       for lps, y in holdout) / len(holdout)
       return min((0.25 * 1.06 ** i for i in range(90)), key=nll)
   

4. Pick where it runs.

If…	Use
Many decision types, no labelled data, accuracy matters more than $2-3 per million decisions	Jev, the most accurate zero-shot option we measured, not the cheapest
You have a few hundred labels and one stable decision	A small LLM or fine-tuned encoder, calibrated
Data residency or on-prem is required	Self-hosted (jeff / GLiFormer, a fine-tuned encoder, or guided decoding on an open model); its raw calibration was already ahead of Jev's in our test
Decisions are rare and the answer needs reasoning or prose	Keep the LLM

One budget note, from the name: TypeSafe named the model after William Stanley Jevons, whose paradox is that cheaper resources get used more, not less. At frontier-model prices you'd never evaluate every frame or keystroke; at $18.57 per million decisions, now a real measured number, you still might. Track cost and volume per decision point regardless of which option you pick.

The one-line version: Jev isn't cheap and isn't best-calibrated out of the box, but it is the most accurate option that needs no labels. Pay for that specifically, or build the free version and spend an afternoon calibrating it.

---

Method, data and limitations

Run 1: cost, latency, accuracy (2026-09-21). 200 articles from the AG News test split, 50 per class, seed 20260921. Five configurations of three OpenAI models, one call per case, 1,000 calls, zero errors. gpt-5.6-luna was called with default settings (no reasoning-effort parameter); its output ran 11–82 tokens per call, mean 14, for a ~5-token JSON answer, and that is included in its cost. Prices: OpenAI Standard tier, verified the same day on OpenAI's pricing page. Latency is client-side from India, network included.

Configuration	Accuracy (95% Wilson CI)	p50	p95	$ per 1M
gpt-4.1-nano, 1 token	71.0% (64.4–76.8)	621 ms	893 ms	$16.30
gpt-4.1-nano, JSON	83.0% (77.2–87.6)	621 ms	996 ms	$20.74
gpt-4o-mini, 1 token	83.0% (77.2–87.6)	675 ms	870 ms	$24.45
gpt-4o-mini, JSON	82.0% (76.1–86.7)	704 ms	1,108 ms	$31.13
gpt-5.6-luna, JSON	82.0% (76.1–86.7)	1,124 ms	1,579 ms	$53.95

Run 2: calibration (2026-09-21). Same 200 test cases plus a disjoint 200-case holdout (seed 20260922), three prompt variants × two models, 2,400 calls. The design (variants, holdout, metrics, the Fisher test) is written in the script's docstring and was fixed before the calls were made. ECE uses ten equal-width bins. gpt-4.1-nano with fixed letters scored 71.0% in run 1 and 69.5% in run 2 on identical inputs at temperature 0: the API is not fully deterministic, and differences of a point or two between runs are noise.

Run 3: the real Jev API (2026-09-21). Same 200 test cases plus the same disjoint 200-case holdout, against https://api.typesafe.ai/v1/systemone, model jev-latest, one choice question per call, 400 calls total. Same accuracy/ECE/temperature-scaling method as Run 2, fit on the holdout and evaluated on the test set, written in the script's docstring before it ran. We deliberately did not run a concurrency/throughput sweep against this endpoint (it is TypeSafe's shared production service, not ours to load-test), so Jev's latency figures are p50/p95 from the sequential run only, comparable to the p50/p95 figures reported for the other backends.

Everything is downloadable. Results: benchmark summary · benchmark calls · calibration summary · calibration calls with all four logprobs · Jev summary · Jev calls · Jev calibration summary · Jev calibration calls. Inputs: 200 test cases · 200 holdout cases · label descriptions. Code, including every prompt verbatim: bench-decision-layer.py · calibrate-decision-layer.py · calibrate-jev.py.

What is still third-party. The 40–200× speed multiplier (TypeSafe's own laptop-measured figure), the jeff project's separate AG News comparison sample, and an independent pre-registered test (~9,750 calls) for calibration by question type across noul/choice/score, the cross-task rank reversal against Claude Haiku 4.5, and the abstention result. We did not reproduce those. Everything else Jev-related in this post (accuracy, cost, latency, and choice-question calibration) is our own measurement against the real API.

Limitations. One dataset, one task, four classes; news-topic classification is easier and cleaner than most production decisions. Two hundred cases give accuracy intervals of about ±6 points and make small bins noisy (several low-confidence bins above have n < 10). Temperature scaling was tested on data from the same distribution as the holdout; it will not survive distribution shift without refitting. Jev's 87.5% (Run 1 arithmetic) and 88.0% (Run 3 calibration run) accuracy figures come from two independent 200-case runs on the same test set, a point apart, consistent with the run-to-run noise we also saw on gpt-4.1-nano. Jev is a week old and in early access; TypeSafe expects prices to fall, and this measurement is a snapshot from 2026-09-21. This analysis was written up the same day the calls were made.

Written by Vikas Chamarthi, founder of NavyaAI, which does LLM inference cost and architecture audits. Benchmark code, data and analysis are ours; prose was drafted with AI assistance and reviewed by the named author. We have no commercial relationship with TypeSafe.

Sources: TypeSafe launch post · TypeSafe docs · Independent pre-registered test, Primeline · jeff, self-hosted System One · Hacker News discussion · Sean Goedecke on structured output · OpenAI API pricing