Not Everything Needs an LLM: A Practical Guide to Decision Models
PostHog's engineers say a small typed model should answer many yes/no questions before an LLM is called. This guide lays out the four patterns they describe, compares each with PostHog's own code and docs, adds what Hacker News objected to, and does the threshold and cost arithmetic so you can decide for your own stack.

There are no affiliate links in this guide.
| Post by | Jina Yoon, PostHog, Nov 9, 2026 |
| Cost | Jev API: $0.042 per 1M input tokens, output free |
| Runs on | Hosted API, or your own GPU (Jeeves, Kev) |
| Catch | Self-reported numbers; thresholds do not transfer |
| Evidence | Post, six pull requests, code, docs, Hacker News |
Use a cheap typed model for narrow yes/no gates, routing and reranking, and keep LLMs for reasoning. PostHog's code backs the pattern, but its headline ratios are best cases and JAVIS ran none of it.
PostHog's own rule: set the threshold based on which mistake costs you more.
What it is. On Nov 9, 2026 PostHog published "Not everything needs an LLM" by Jina Yoon. A decision model takes some state plus typed questions (yes or no, pick one, score) and returns a typed answer with a probability instead of prose. The post describes four uses at PostHog: gate expensive LLM calls, route a request to the cheapest capable model, rerank search results, and replace fixed rules.
Why it matters now. The post reached 187 points and 75 comments on Hacker News (story 50535740) by Nov 21, 2026. PostHog also published its own decision model, Jeeves, under an open license (repository created Sep 29, 2026), and its code and pull requests are public, which lets you compare the post with what actually shipped.
The catch. Three things. First, the 444.6x cheaper and 193.6x faster figures come from TypeSafe's own workflow tests, and TypeSafe's blog says they are on the higher end of real-world gains. Second, the production examples ran on a self-hosted model called JevK5 and then on Jeeves behind PostHog's own gateway (pull requests below), not on the hosted Jev API the post links to. Third, a score of 0.85 meaning "right 85 percent of the time" only holds if the model is calibrated for your task: PostHog itself lowered its personal-data bar from 0.8 to 0.4 after the older model caught 29 percent of answers that named a person.
Not a hands-on test: JAVIS did not install or run Jev, Jeeves, Kev or any PostHog feature and did not test PostHog's decision models. The evidence is PostHog's own post plus the Hacker News discussion, both self-reported, cross-read against PostHog's public pull requests, source files and docs, TypeSafe's docs and the model repositories, all as of Nov 21, 2026.
What you need to know in thirty seconds.
| Fit score part | Points of 20 |
|---|---|
| Usefulness | 16 |
| Evidence depth | 14 |
| Tooling maturity | 11 |
| Reproducibility | 15 |
| Risk clarity | 12 |
| Total | 68 of 100 |

Not a chatbot on a short leash: a model that scores a fixed set of answers.
TypeSafe, the company behind the hosted Jev model, defines the idea in its docs. You send a state (text, JSON or images) and typed questions. Each question is evaluated in parallel against the state and in isolation from the others. A Noul question asks whether a statement is true and returns a value from 0 to 1. A Choice question picks one option from a list and returns probabilities. A Score question rates the state on a rubric. A fourth type, Box, finds where something is in an image and needs Jev 1.14, released Oct 14, 2026.
The default limits matter for what you can ask: 64k tokens for the whole request and 32k for the state plus any one question. A long-context header raises both to 262k tokens, at the same price per token, but TypeSafe says it is slower and less accurate. TypeSafe also states that Jev is not trained on customer requests or responses.

Is it an LLM?
The first objection on Hacker News was about the name. One reader asked what a decision model is if not an LLM with instructions to limit its replies (50538899). Another answered that it is an LLM in the sense BERT is, neither generative nor instruction-tuned, and that the framing is partly marketing (50539552). A third said models like Jev are language models with a different architecture built for one-shot selection (50545010).
The primary pages support the reader. PostHog's Jeeves model card on Hugging Face lists Qwen3.5-9B as its base model, with a LoRA adapter and a pointer head on top. So at least that decision model is a fine-tuned language model whose output is constrained to the offered options. In practice the useful difference is the interface, not the label. The answer is a probability over options your code defined, produced in one pass instead of generated text your code has to parse.


Gate, route, rerank, replace: each with the number PostHog uses and what its code shows.
Swipe sideways to see every column.
| Pattern | PostHog example | Bar | What the code or docs show |
|---|---|---|---|
| Gate expensive calls | Personal-data check on Replay Vision answers | 0.4 on either of two questions | Pull request 120387 (merged Oct 20, 2026) runs two questions in one request and flags at 0.4 or higher |
| Gate expensive calls | Slack bot answering untagged channel questions | 0.85 on both questions | Source file sets 0.85, two yes/no questions, a 5 second timeout; a miss costs the author one mention |
| Route | Slack bot picks a model for a new task | None (one choice question) | Source file holds the instructions: cheapest option likely to finish on the first try |
| Rerank | Replay Vision search results | None (reorders top 20) | Pull request 107372: ranking score 0.57 to 0.72 on 20 queries; falls back to the old order after 2 seconds |
| Replace rules | Chart choice for SQL results | Not stated | Pull request 108065 calls it an experiment behind a flag and needs organization consent |
1. Gate: personal data in session summaries
Replay Vision watches a session recording and writes a summary. PostHog's docs say every scanner answer is checked for personal data the prompt did not ask for, and an answer that still contains it after one rewrite fails and is not saved. The post describes the first filter: two narrow yes/no questions, one about a named person and one about contact details or an ID. If either scores 0.4 or higher, Gemini rewrites the summary and it is checked again. PostHog says 98.7 percent of summaries skip the rewrite in production.
The pull request adds the context the post leaves out. Before the change, the older model at a 0.8 bar caught under a third of answers that named a person. Splitting the check into two narrow questions gave about a tenth of the false flags of a single question. Lowering the bar to 0.4 is the real tuning step: the pull request says the model scores most real names between 0.4 and 0.8, so the old bar missed them.


2. Gate: the Slack bot that answers without a tag
The second gate uses a higher bar on purpose, because a wrong answer interrupts a public conversation while a miss only costs someone one @PostHog. The source file says both questions must clear 0.85 and that the value was tuned for precision over recall against an evaluation file in the repository. PostHog's docs describe the user-facing side: by default the bot asks the author first in a prompt only they can see, and a user can switch to answering automatically or never. The docs also say behavior is still moving in the beta.

3. Route: the cheapest model that will probably finish
For a new Slack task the bot first checks whether the message names a model. If not, one choice question picks from a short list, where each option carries a note about what that model suits. The instructions in the source file tell the model to prefer the cheapest option likely to finish on the first try and list when to pick a smarter option (new features, refactors, bug causes, unclear requests) or a faster one (a direct question, one query, a small clear edit). The post's own caveat is that thresholds and behavior change between models and between versions of the same model.

4. Rerank: let embeddings find candidates, let the model judge matches
Embedding search finds items that share a topic with the query, which is not the same as matching it. PostHog sends the top 20 results to a decision model that returns a match probability for each and sorts by it. In the pull request, an offline evaluation on 20 queries graded blind moved the ranking score (nDCG@10) from 0.57 to 0.72. That is a gain of 0.15 points, or 26 percent in relative terms, which matches the post's "about 25%". The design also sends two parallel requests of 10 instead of one of 20, because the model reads every question against the whole request and cost grows with the square of the candidates. The rerank never blocks search: a timeout or a gateway error returns the embedding order after at most 2 seconds.
Where PostHog says not to use one
Two negative results are in the post. Chaining two questions to pick sessions worth watching, then a reason, sent large hourly requests of up to 32 sessions and 12 to 38 thousand tokens each and, PostHog says, blocked other products' calls for hours. And ranking an internal pull-request review agent's findings with five yes/no questions was inconclusive, with an AUC of 0.52 (0.5 is a coin flip). The post's conclusion: decision models judge what is in front of them in a single pass, so leave multi-step reasoning to LLMs. A highlights feed appears in a public pull request (114045, Oct 8, 2026), but the flooding incident and the 0.52 figure have no public source beyond the post, so both rest on the post alone.


Which statements in the post a second source confirms, and which rest on the post alone.
Swipe sideways to see every column.
| Claim in the post | Second source | JAVIS status (Nov 21, 2026) |
|---|---|---|
| Up to 444.6x cheaper and 193.6x faster than frontier models | TypeSafe blog: from its own workflow tests; "on the higher end of real world gains"; reference answers come from two frontier models and the tests were built by its own team | Real figure, but a vendor best case; not PostHog's measured cost |
| 98.7% skip the LLM call in production | No public source; the pull request linked from the post does not contain it | Post only |
| Two narrow questions cut false flags to about a tenth | Pull request 120387: the split version had the fewest false flags, about a tenth of the single question | Confirmed by PostHog's own pull request (LLM-labelled sample) |
| 0.4 for personal data, 0.85 for the Slack bot | Pull request 120387 and the Slack source file (0.85, both questions) | Confirmed in code |
| Routing picks the cheapest option likely to finish first try | Model router source file | Confirmed in code; routing quality not shown |
| Rerank improved ranking about 25% on 20 queries | Pull request 107372: nDCG@10 0.57 to 0.72 | Confirmed (26% relative), small sample |
| SQL chart picker uses a decision model now | Pull request 108065 is titled an experiment, behind a flag, organization consent required | Partly: status in production not shown |
| Highlights feed flooded the model server; PR review ranking AUC 0.52 | No public source | Post only |
| Jeeves is now the default decision model at PostHog | Pull request 120316 (Oct 20, 2026): shared default moves from JevK5 to Jeeves | Confirmed in code |

One more reading of the cost ratio. TypeSafe's blog puts the input prices of the frontier models it compares at $0.20 to $10 per million tokens, against $0.042 for Jev. On input price alone that is 4.8x to 238x cheaper (0.20 / 0.042 and 10 / 0.042). Reaching 444.6x needs the output side too, where Jev output is free and the blog gives LLM output as about five times the input price. So the ratio depends on how long the answers are, which is exactly where a typed answer wins and a prose answer loses.
What a threshold means, and when a gate pays for itself.
Illustrations built from the cited figures. The cost examples use stated assumptions and the Jev API price; PostHog runs its own models behind its gateway, so its cost is not billed this way.
What a bar means if the model is calibrated
The post's rule: in a calibrated model, answers that score 0.8 are right about 80 percent of the time. So for items that score exactly at your bar, the share of wrong answers is one minus the bar. Real items score above the bar on average, so this is the worst case at the edge. Hacker News readers asked whether 0.85 really is 85 percent (50541184, 50541412), and TypeSafe's docs back the caution: its confidence number is a different quantity from the probability that the chosen answer is correct, and the right threshold depends on your domain and data.
Two bars side by side. At 0.4 a flag at the edge is more likely wrong than right, which is fine because the cost of a false alarm is one more LLM call, while a miss stores personal data as clean. At 0.85 on both Slack questions the chance both are right at the edge is 0.85 x 0.85 = 0.72 if the two answers are independent and calibrated, so about 278 of 1,000 edge cases would be answered wrongly. JAVIS assumed independence here, which real questions rarely have, so treat 278 as an upper bound for the edge.
Why recall decides the privacy case
The post is blunt that a false negative stores personal data as clean. Take 1,000 answers that truly name a person. At the 29 percent recall PostHog measured for the older model at 0.8, about 710 pass through unflagged. At 97 percent recall for Jeeves at 0.4, about 30 do. Both are measured on a sample of 121 positive answers labelled by LLM subagents, so the real figure can differ, and a gate with 97 percent recall still leaks.
When a gate pays for itself
A gate is worth running when its cost per item is below the share it lets you skip times the cost of the LLM call it avoids. Assume an 800-token summary and two questions on the Jev API: 800 x 2 x $0.042 / 1,000,000 = $0.0000672 per item, assuming both questions are billed on the full state (the worst case; the docs say the state is read once). An LLM call on the cheapest input price in TypeSafe's range, $0.20 per million, costs 800 x $0.20 / 1,000,000 = $0.00016 for input alone.
Swipe sideways to see every column.
| Share skipped | LLM calls left per 10,000 | LLM call must cost more than | Pays at $0.00016 per call? |
|---|---|---|---|
| 98.7% (PostHog figure) | 130 | $0.0000681 | Yes |
| 90% | 1,000 | $0.0000747 | Yes |
| 50% | 5,000 | $0.0001344 | Yes, narrowly |
| 10% | 9,000 | $0.000672 | No |
The break-even rule is cost per item < skip rate x LLM cost, so the LLM call must cost more than the gate cost divided by the skip rate. At these prices the dollars are small either way: 10,000 summaries a day cost about $0.67 for the gate and $1.60 for input-only LLM calls with no gate, or about $0.69 with the gate and 130 remaining calls. The gain at this scale is latency and capacity (fewer slow calls, fewer rate-limit hits), not the invoice. The dollar case grows with volume and with expensive models.
The post says Jev. The pull requests show a moving target: JevK5, then Jeeves, with fallbacks everywhere.
The model changed under the post. The post names Jev and links TypeSafe. The pull requests say the rerank ran on JevK5, a decision model self-hosted behind PostHog's AI gateway in the US and EU, and that on Oct 20, 2026, 20 days before publication, the shared default and the personal-data check moved to Jeeves, to retire JevK5. The post does mention Jeeves as the default for any decision call that does not name a model. So the lesson for a reader copying the pattern: the 0.4 and 0.85 bars belong to specific models, and PostHog says its error-tracking bars changed between v1 and v2 of the same model.
Shadow mode before enforcement. The error-tracking pull request (117014, Oct 14, 2026) lets a Jeeves judge propose merging near-duplicate errors, with the cutoff and a shadow mode controlled by a feature flag payload. In shadow mode every judged pair is recorded with its score but never merged, so decisions can be reviewed first. A judge that cannot answer leaves things unmerged.
Failure falls back to the old behavior. The rerank returns the embedding order on timeout or gateway error, the untagged-question gate treats a refusal as no answer, and the personal-data check returns unknown on a gateway error. That is a design habit worth copying: the cheap model is an optimization layer, never the only path.
The models, and how they compare as self-reported
Jeeves is a 9B-parameter model (Qwen3.5-9B with a LoRA adapter and a pointer head) that answers directly when confident and writes a short reasoning chain first when not. The README says it takes about 20 milliseconds per request without thinking and a 1.2 second median with thinking on one H100 in FP8, and needs Python 3.12 and a CUDA GPU (or an Apple Silicon Mac with 48 GB or more). The repository was created on Sep 29, 2026, has 470 stars, no releases and an MIT license, while its Hugging Face card carries an Apache-2.0 tag, so read the license file before relying on either. Kev, a separate project by Jared Palmer that Jeeves credits as inspiration, has about 10,100 stars, an Apache-2.0 license and a Kev 1.0 release.


A hosted API, two open-weight models, or no model at all: pick by the cost of a wrong answer.
Swipe sideways to see every column.
| Option | What it is | Facts (Nov 21, 2026) |
|---|---|---|
| Jev API (TypeSafe) | Hosted decision model: Choice, Score, Noul, Box questions | Jev 1.14 and 1.13 both $0.042 per 1M input tokens, output free; 64k per request and 32k per question by default; 262k with a long-context header, slower and less accurate |
| Jeeves (PostHog) | Open 9B decision model that reasons when unsure | MIT repository, Apache-2.0 tag on the model card, 470 stars, no releases; about 20 ms without thinking and 1.2 s median with it on one H100 (self-reported); you supply the GPU |
| Kev (Jared Palmer) | Open family of Jev-like models, 0.8B to 27B | Apache-2.0, about 10,100 stars, Kev 1.0 release; API matches TypeSafe System One per its README |
| Rules, regex, plain code | Deterministic functions | Free, instant, exact; commenters on Hacker News argue many tasks never needed a model (50552687) |
| Embeddings plus a trained classifier or reranker | Smaller supervised models | A reader says trained rerankers already return calibrated scores and are smaller and faster (50542116); PostHog uses embeddings first and a decision model second for search |
| A general LLM with structured output | The default most teams start with | Handles multi-step reasoning and long context; PostHog keeps it for open-ended work |


Hacker News: sympathy for the idea, objections on definitions, privacy and thresholds. The sample is small and self-selected.
The thread (story 50535740) showed 187 points and 75 comments on Nov 21, 2026. Four comments worth acting on:
Asks what a decision model is, exactly, and whether it is just an LLM told to limit its answers. A reply calls it an LLM that is not generative or instruction-tuned.
Says PostHog should not use these models on customer data at all. A reply notes the tickets were not supposed to hold personal data in the first place (50544015).
Argues that everyone should use the most expensive LLM unless they know how to run offline and online evals, because weak evals leave a product broken for the p90 customer and the fat tail is hard to sample.
Says not everything needs a decision model either: if two lines of code can answer, a service request is slower than the code.
Other readers questioned whether the router can know newer models from a short note (50540197), called a model that cannot reason limited to basic decisions (50541862), and defended the idea: the answer cannot leave the offered options (50540370), and a decider is closer to producing one token than decoding JSON (50542906).
Swipe sideways to see every column.
| Objection | What the primary pages show | Still open |
|---|---|---|
| It is just an LLM | Jeeves is a Qwen3.5-9B fine-tune; Jev is described by TypeSafe as a different model family | Whether the label matters for your decision |
| Personal data should not go through a model | PostHog says the gate exists to catch personal data before it is stored; Replay Vision sends recordings to a Gemini model; features need organization consent per the pull requests | Recall is 97% on a small LLM-labelled sample, so some data still passes |
| 0.85 is not really 85% | TypeSafe docs separate confidence from correctness; PostHog retuned from 0.8 to 0.4 for names | Calibration must be measured on your own data |
| Evals are hard | PostHog used blind-graded samples and a shadow mode; samples were 20 queries and 121 positives | No public eval set for the production gates |


Where this guide, and the pattern, stop.
- Self-reported evidence. The post, the pull requests and the Jeeves README all come from PostHog. The ratios come from TypeSafe. JAVIS did not run any model.
- Small samples. 20 queries for the rerank, 45 clean and 121 positive answers for the personal-data check, with labels made by LLM subagents. PostHog says the new criteria wording partly explains the gain.
- Moving targets. The post's author expects it to be obsolete in two months. The default model changed 20 days before publication and Jev 1.14 appeared on Oct 14, 2026.
- Thresholds do not transfer. PostHog reports different bars for different tasks, models and versions of the same model. Copying 0.4 or 0.85 without your own labelled data is the main trap.
- Short context. Jev's default limit is 64k tokens per request; the long-context mode is slower and less accurate.
- Licensing detail. The Jeeves repository says MIT and the model card tags Apache-2.0. Check the license file before shipping.
- Privacy. A gate with imperfect recall is not a privacy control. Treat it as one layer next to consent, redaction and retention rules.
- Not covered: hands-on tests, latency on your hardware, accuracy on your data, other decision models, and legal advice.
A short decision list.
You call an LLM many times a day for yes/no or pick-one judgments on short input, can label a few hundred real examples, and have a fallback if the cheap model fails or times out.
A rule would do, the task needs several reasoning steps or long documents, you cannot measure errors on your own data, or you plan to use the score as your only privacy control.
Five actions.
No affiliate links; the only list price is TypeSafe's.
The Jev API is priced at $0.042 per million input tokens with free output, per TypeSafe's docs on Nov 21, 2026. Jeeves and Kev are open-weight, so their cost is your hardware. PostHog's post is written by a PostHog engineer and its Jeeves model is also a PostHog project, so the author has an interest in the approach. JAVIS has no relationship with PostHog or TypeSafe, and this guide contains no affiliate links.
Building with model calls in production? Read these next.
Questions people ask about decision models.
What is a decision model?
A model that takes a state and typed questions and returns a typed answer with a probability, such as yes or no with 0.99, instead of free text. TypeSafe's Jev, PostHog's Jeeves and the open Kev family are examples.
Is it still an LLM?
Often built on one. Jeeves is a fine-tune of Qwen3.5-9B. The practical difference is the constrained, probability-bearing answer, not the absence of a language model.
What does a score of 0.85 mean?
If the model is calibrated, about 85 percent of answers scored 0.85 are right. TypeSafe's docs warn that its confidence value is not the chance the answer is correct, and PostHog reports that bars differ by task and model, so measure it on your data.
How much does Jev cost?
$0.042 per million input tokens, output free, for Jev 1.14 and 1.13, per TypeSafe's docs on Nov 21, 2026.
Can I run one myself?
Jeeves and Kev publish open weights; Jeeves needs a CUDA GPU or a large-memory Apple Silicon Mac per its README. JAVIS did not run either.
Did JAVIS test PostHog's decision models?
No. This guide is built from PostHog's post, its public pull requests, source files and docs, TypeSafe's docs, the model repositories and the Hacker News discussion, all as of Nov 21, 2026.
Where this came from, and when.
Post: newsletter.posthog.com/p/decision-models, by Jina Yoon, dated Nov 9, 2026 on posthog.com (the Substack page shows Nov 10), as of Nov 21, 2026.
PostHog pull requests: 120387, 120316, 107372, 117014, 114045, 108065, all as of Nov 21, 2026.
PostHog source and docs: untagged_question.py, model_router.py, Replay Vision docs, Creating scanners, Slack app docs, as of Nov 21, 2026.
Models: TypeSafe models, TypeSafe confidence, TypeSafe blog on Jev, PostHog/jeeves, Hugging Face PostHog/jeeves, jaredpalmer/kev, as of Nov 21, 2026.
Hacker News: story 50535740 with comments 50538899, 50539552, 50545010, 50538646, 50544015, 50541184, 50541412, 50538986, 50552687, 50540197, 50542116, 50541862, 50540370 and 50542906, as of Nov 21, 2026.
Images: screenshots of the pages above taken by JAVIS on Nov 21, 2026; diagrams, charts and the timeline drawn by JAVIS from the cited numbers. The Fit Score is editorial judgment, not a benchmark.
Not covered: hands-on tests, accuracy on your data, latency on your hardware, other decision models and legal advice.



