Jev and typed decisions: what they guarantee, and what they do not

The claim that drew everyone's attention to TypeSafe's Jev was that it does not hallucinate. That claim is true, in a precise and narrow sense, and the narrowness is where all the engineering judgement lives.

I built the same mechanism into Antan IRM and measured it against four candidate uses inside the product. One shipped. Three did not. The three that did not are the more useful story, so this piece is mostly about those, with the numbers attached.

What the mechanism actually is

A normal language model answers by writing text. You then read that text back and pull a value out of it. When the text arrives malformed, or carries a value nobody planned for, your code falls back to a default.

A System One model does not write the answer. You declare the permitted answers as part of the question, the model produces a single decision, and you read its scores across those permitted values alone. Jev exposes three shapes of question at POST /v1/systemone: Choice picks one option from a set you define, Score rates against levels you define, and Noul returns the probability that a yes or no statement is true. TypeSafe's published figures put it at 70 to 500 milliseconds a call, at $0.042 per million input tokens with output unmetered.

You do not need their API to get the behaviour. The same mechanism works against a local model: enumerate the options, generate exactly one token, and restrict the probabilities you read to the tokens standing for those options. That is what Antan IRM does, on hardware I control, and every figure below is measured from that.

The guarantee this buys is real and worth stating exactly: an answer outside your permitted set cannot be produced. Not filtered out afterwards. Not validated away. There is no token for it to land on. That is a guarantee about the shape of the answer. It says nothing whatsoever about whether the answer is right.

Where it earns its place in security

The bug class it removes is one I have found in real systems more than once, and it is nastier than it looks. Here is a real line from an earlier version of my own vendor-matching code:

bool(j.get("is_same_entity", True))

That reads: if the model's answer could not be parsed, assert that the two organisations are the same. A failure to answer became a confident yes, stored in the same field, with the same type, in the same format as a real verdict. Nothing downstream could tell them apart and no alert fired.

That is what makes it a security problem rather than a quality problem. A silent default is an undetectable failure, and an undetectable failure in an identity check is exactly the sort of thing an attacker gets to rely on.

This use survived measurement. Forty vendor pairs, deliberately half hard negatives: similar or identical trading names in a different country or a different industry. Accuracy 1.0, zero false positives, and the separation was total rather than marginal, with the highest score on a false pair at 0.0000016 and the lowest on a true pair at 0.997. That one shipped, and it is live.

Where it did not survive contact: decision analysis

Three candidate uses failed, each for a different reason, and each reason generalises.

It lost to a boring incumbent

I tried it for routing a question to the right specialist agent: a nine-way classification, the textbook case for Choice. Over sixty labelled questions it scored 0.833, or 0.867 with a debiasing pass that evaluates the options in both orders and averages.

The incumbent it was replacing, a plain language model writing JSON with a keyword ladder behind it, scored 1.000 on the same sixty.

The new mechanism was about a hundred times faster: 30 milliseconds against 2,921. It did not ship. Being faster, cheaper, and more principled is not the same as being more accurate, and on a classification that steers the rest of the analysis, accuracy is the only currency that counts.

The rating depended on how I listed the options

This is the finding that matters most for decision analysis, and I have not seen it discussed anywhere.

Score asks the model to rate something against levels you define. I used it to rate contract clauses across low, medium, high and critical. Then I listed those four levels in the opposite order and asked again.

The band moved on 25 of 40 clauses.

A rating that changes depending on the order you wrote the options in is not measuring the clause. It is partly measuring your prompt layout. For a classification you might tolerate that; for a risk rating that feeds a decision, it is disqualifying. It is also not something calibration can fix, because rescaling a distribution provably cannot reorder it.

The confidence was not honest

The rating claimed near-certainty and was exactly right on 45 per cent of clauses. Measured as expected calibration error, which is the average gap between claimed probability and observed frequency, that is 0.4745: the confidence is off by roughly 47 percentage points.

Fitting a single temperature to correct it, using leave-one-out so that no clause is ever scored by a correction fitted on itself, brings it to 0.2262. Better, and still far too high to display. The fitted temperature was 11.02, meaning the distribution has to be flattened elevenfold before the confidence means what a reader assumes it means, and at that point there is very little distribution left to show anyone.

The important part is that this is not a quirk of reading a general model's logits. Published figures for purpose-built typed-decision models sit in the same territory. The open laya-typed-decisions model reports an ECE of 0.213 on its own benchmark, against TypeSafe Jev 1.13.0's published 0.144 on the same one, and that project's documentation states plainly that both its checkpoints are over-confident as shipped. Its base checkpoint's raw ECE is 0.466, which is within a whisker of my 0.4745.

The conclusion I draw is not that one backend is better calibrated than another. It is that calibration costs labelled data whichever backend produces the distribution, and nobody is handing it to you for free.

Always run the stupid baseline

My contract rating scored 0.950 "within one level" on a four-level scale. That sounds strong.

Then I ran a predictor that reads nothing at all and always answers "medium". On the same forty clauses it scored 0.775.

So the entire value of the model over a constant was 0.175. On a short ordinal scale, adjacent-level credit is generous to begin with, and a headline percentage that nobody has compared against a constant is not evidence of anything. This costs about four lines of code and it has changed what I shipped more than once.

Two practical constraints worth knowing before you plan around this

The model's first generated token has to be the answer. This quietly rules out most current models. A reasoning model opens a chain-of-thought block first; a Harmony-formatted model emits a control token first. Neither ever produces your option token first, so the mechanism reads nothing usable. In my testing, one 16B reasoning model produced zero usable answers across 140 calls, and the 26B model that serves the rest of the product produced zero across 60. "It runs on any open model you already have" is not true. Test a candidate before you build a plan on it.

The questions cannot see each other. Each is answered independently, so a field whose correct value depends on another field's answer does not belong here. TypeSafe say this about their own model, and it is a property of the mechanism rather than a defect. Ordinary generation, which can look back at what it already wrote, beats this approach on exactly that shape of problem.

On speed: I measured twelve independent questions against a shared context at 331 milliseconds one at a time and 234 milliseconds concurrently, a 1.41x improvement rather than the order of magnitude the public demonstrations report. The difference is that my inference server deliberately runs a single slot, because four slots made a 9 GB model occupy 19.5 GB of video memory. The larger saving was elsewhere and was already working: re-reading a shared context costs 41 to 45 milliseconds cold and 27 milliseconds once cached. The dramatic numbers are real, but they are bought with memory, and that is a capacity decision rather than a free lunch.

Where I would actually use it

High-volume, genuinely independent questions, where you act on the distribution rather than display it, and where you have enough labelled examples to state an error rate: triage, routing, relevance filtering, identity matching. That is a real and valuable category, and the schema guarantee makes the failure modes honest ones.

I would not use it, today, for anything whose confidence number a board will act on. In cyber risk that is the harder half of the problem. A rating that is merely well-formed but confidently wrong is worse than no rating, because it travels further before anyone questions it.

Which is why, in Antan IRM, the contract risk rating is calculated on every analysis and deliberately not displayed. The tool says the rating is being withheld pending calibration, and it will keep saying that until the numbers above clear a bar I set before I ran the test rather than after.

The type safety is free. The decision quality is not.