We Ran the Open-Source Jev Alternative With No Cloud GPU: 75% Accuracy, Confidence That Was Already Calibrated

We ran the open-source alternative to TypeSafe's Jev on an ordinary laptop — no cloud GPU, no rented instance — and measured it against the same 200 cases as our Jev benchmark. 75.0% accuracy. Calibration error of 0.053 with no fitting at all. If you'd rather rent EU cloud hardware instead, we worked out what that costs too, and show the arithmetic.

Our last post tested the do-it-yourself imitation of Jev — a general-purpose LLM forced to answer in one token. This one tests something different: an actual open-source, self-hostable model built to speak Jev's own API. If your reason for reading either post is "we cannot send this data to a US-hosted API," this is the one that answers your question.

The verdict

1. You do not need a cloud GPU. We ran this on a laptop. It worked.
2. Its confidence needed no fixing. Unlike the constrained-token LLM in our last post, which was badly overconfident until calibrated, this model's confidence was already trustworthy out of the box.
3. The accuracy trade is real. 75% against Jev's published 90.5% on the same task. Decide whether that gap matters for your decision before you commit to either direction.

What jeff is

jeff is an MIT-licensed open-source project that reimplements Jev's own API — the same state + typed questions shape, the same noul/choice/score primitives — backed by GLiFormer, a 400-million-parameter open model. You run it yourself: pip install, download the model, start a server. Nothing calls out to TypeSafe, or to anyone.

# Same request shape as Jev itself. Point at your own server instead of theirs.
resp = httpx.post("http://localhost:8000/v1/systemone", json={
    "model": "jev-latest",             # jeff serves this as an alias
    "state": article_text,
    "questions": {"topic": {
        "type": "choice",
        "instructions": "Classify this news article into one topic.",
        "criteria": {"world": "...", "sports": "...",
                     "business": "...", "scitech": "..."},
    }},
}, headers={"Authorization": "Bearer devkey"})

answer = resp.json()["answers"]["topic"]
topic, confidence = answer["choice"], answer["confidence"]

That API compatibility is the whole point: code written against Jev runs against jeff with a one-line URL change.

What we measured, and how

We had already published scripts/bench-decision-layer.py and 200 human-labelled test cases (AG News, 4 classes) for the last post. Same cases, same holdout, same method — the only thing that changed is which model answers.

Setup. Cloned jeff (MIT license, verified), checked every pinned dependency for known vulnerabilities before installing anything. Two flagged as "malicious" — both turned out to be unrelated npm packages that happen to share a name with the real PyPI packages (pydantic, uvicorn); confirmed clean by querying OSV.dev directly, scoped strictly to the PyPI ecosystem. Installed into an isolated Python 3.12 environment, downloaded the model (2.2GB), ran the server locally with JEFF_DEVICE=mps — Apple Silicon's GPU cores, not a dedicated graphics card, not a cloud instance.

One honesty fix before running anything. Our benchmark script's jev backend defaults to calling TypeSafe's real API and would have labelled these rows backend=jev regardless of where the request actually went. Given what happened with our last post, that is exactly the kind of thing that must not happen quietly. It now refuses to run against any URL that isn't typesafe.ai unless you pass an explicit relabel — --jev-label jeff-local — so a self-hosted run can never be mistaken for a call to the real API in the published data.

TYPESAFE_API_KEY=devkey TYPESAFE_BASE_URL=http://localhost:8000/v1/systemone \
  python scripts/bench-decision-layer.py \
  --cases data/decision-cases.jsonl --labels data/decision-labels.json \
  --task "Classify this news article into one topic." \
  --backends jev --model jev-latest --jev-label jeff-local

Results: accuracy and calibration

	Accuracy	ECE (raw)	ECE (temperature-scaled)	Right at ≥0.9 confidence	Right at <0.8 confidence
jeff-local (measured, this post)	75.0% (n=200)	0.053	0.056	100.0% (n=15)	66.4% (n=137)
gpt-4.1-nano, 1 constrained token (measured, last post)	71.0%	0.258	0.074	74.3% (n=171)	35.0% (n=20)
gpt-4o-mini, 1 constrained token (measured, last post)	83.0%	0.175	0.055	82.7% (n=197)	50.0% (n=2)
Jev (third-party, jeff project's own AG News run)	90.5%	—	—	—	—
jeff (third-party, jeff project's own AG News run)	75.5%	—	—	—	—

Two things stand out.

Our 75.0% lands almost exactly on jeff's own published 75.5%. Different sample, independently run — that is what an honest reproduction is supposed to look like, and it is a good sign for the reliability of both numbers.

The calibration story is the opposite of what we found with the DIY alternative. In the last post, forcing a general LLM to answer in one token produced wildly overconfident results — 95-99% average confidence at 70-83% accuracy — that needed a fitted temperature parameter to become trustworthy. jeff's confidence needed nothing. ECE 0.053 raw is already in the "usable for gating" range, and fitting temperature on a separate 200-case holdout barely moved it (0.056 — statistically indistinguishable, if anything slightly worse, which is what you'd expect when there is nothing left to fix and the fitting is chasing holdout noise).

That is a real advantage of a model purpose-trained to output calibrated decisions over a general-purpose model wearing a decision-shaped disguise. It is also, not coincidentally, the same property TypeSafe claims for Jev itself — the two models were built for the same job.

Throughput: measured on a laptop, not modelled

Concurrency	Requests/sec (measured, M4 Pro, MPS)
1	9.2
2	14.0
4	17.1
8	12.8
16	16.0

p50 latency 236ms, p95 458ms, sequential. That ceiling — roughly 15-17 requests/second peak — is on a laptop with no dedicated GPU, doing nothing to optimise for throughput. For context, jeff's own project notes cite about 50 requests/second per container on a rented L4 in the cloud, a real datacenter GPU: about 3× more throughput than what we got locally.

For most of the decisions this post is actually about — ticket routing, document triage, agent tool-gating — 15-17 decisions per second is 1.3-1.5 million per day from a single machine you already own. That is not a rounding error next to most teams' actual decision volume, and it costs nothing beyond the electricity.

If you do want a cloud GPU: the EU hosting-cost table

Some workloads genuinely need more throughput than one laptop can give, and want it from a provider based in the EU. Here is what that costs, computed transparently rather than measured by us — we did not rent any of these instances.

The arithmetic: jeff's own published throughput on a rented L4 (~50 requests/second) × each provider's own hourly list price, verified directly against their pricing pages on 2026-09-21.

Provider	GPU	$/hour	$ per 1M decisions	Basis
Scaleway	L4	$0.91 (€0.79)	$5.04	EU-native (France); hourly cloud
OVHcloud	L4	$1.00	$5.56	EU-native (France); hourly cloud
AWS eu-central-1 (Frankfurt)	L4 (g6.xlarge)	$1.01	$5.59	US company, EU datacenter; hourly cloud
Hetzner	RTX 4000 SFF Ada 20GB	$0.29-equivalent	$1.61	EU-native (Germany); monthly dedicated server, not hourly cloud, not an L4

Read the Hetzner row carefully before you get excited about it. It is not the same comparison as the other three: it is a physical box you lease by the month (€184/mo, or €1,199/mo for a larger card), amortised to an hourly-equivalent rate for this table, running a different-generation GPU. There is no elastic scaling and no pay-as-you-go. It belongs in this table because "lease a dedicated EU box monthly" is a real option for a team in this position, not because it is a fair apples-to-apples price comparison with the cloud rows above it.

One more honesty check: jeff's own project publishes roughly $2.60 per million requests on their Modal deployment — about half what our arithmetic gives for OVHcloud, Scaleway or AWS. We do not know why without testing it ourselves; Modal's serverless pricing model and jeff's own batching may simply beat a naive on-demand instance. Take both figures as directional, not final, and benchmark your own workload before committing spend either way.

What this means if you cannot send data to a US-hosted API

Jev is a closed, hosted API, served from TypeSafe's own infrastructure in the US. There is no on-prem or self-hosted version of Jev itself, and no published path to one. If your constraint is genuinely "this cannot leave our servers" — a regulated industry, a data-residency clause, an internal policy — that is not a pricing question. jeff, or a model like it, is the only category of answer available to you, not one option among several.

What you are trading for that is fifteen accuracy points against Jev's published number, on this task. Whether that trade is worth it depends entirely on what the decision is and what a wrong answer costs you. A support-ticket router that occasionally sends a ticket to the wrong queue is a different risk than a compliance gate. Measure it on your own labelled data before you decide, the same way we are telling you to measure everything else in this series.

Method, data and limitations

What we measured. 400 calls (200 test + 200 holdout, both from the AG News test split, disjoint, seed-controlled) against a jeff server we ran ourselves, plus a throughput sweep at five concurrency levels. Zero errors. Hardware: Apple M4 Pro, MPS backend, no dedicated GPU. Data: accuracy and calibration summary · all 400 calls · throughput sweep.

What is third-party. Jev's 90.5% accuracy and jeff's 75.5% accuracy on AG News: published by the jeff project, not reproduced by us beyond our own independent 75.0%. jeff's ~50 req/s L4 throughput figure and ~$2.60/1M cost figure: published in the jeff project's own notes. All EU provider prices: each provider's own pricing page, fetched and verified 2026-09-21.

Reproduce it.

git clone https://github.com/logan-markewich/jeff
cd jeff && uv sync --extra dev
uv run hf download knowledgator/gliformer-large-v1 \
  --local-dir models/gliformer-large-v1
JEFF_API_KEYS=devkey JEFF_DEVICE=mps uv run jeff &

python scripts/calibrate-jeff.py \
  --base-url http://localhost:8000/v1/systemone --api-key devkey \
  --out public/data/decision-layer-jeff.csv

Script: calibrate-jeff.py.

Limitations. One dataset, one task, four classes — the same limitation our last post had, for the same reason: it lets every number here sit next to a directly comparable one. Two hundred cases gives accuracy intervals of roughly ±6 points; the confidence-bucket splits above are noisy at the low end (n=15 for ≥0.9 confidence). Our throughput is one laptop under a synthetic concurrent-request pattern, not a production load test, and is not a substitute for measuring your own hardware. The EU cost table is arithmetic on a third-party throughput figure, not a measurement on those specific providers — we say so above, and we mean it: verify it yourself before you commit spend.

Written by Vikas Chamarthi, founder of NavyaAI, which does LLM inference cost and architecture audits, including on-premise and EU-resident deployments. Benchmark code, data and analysis are ours; prose was drafted with AI assistance and reviewed by the named author. We have no commercial relationship with TypeSafe or with the jeff project.

Sources: jeff, self-hosted System One · GLiFormer model card · Our Jev benchmark · OVHcloud GPU pricing · Hetzner dedicated GPU servers · Scaleway L4 GPU instances · AWS EC2 g6 instances