# Evaluating decision models: evidence, rules, and escape options

**Local draft — practical guidance successor; independent scientific/editorial review and browser QA pending; not published.** The accepted Jev prerequisite page is preserved unchanged. Its independent local scientific integration and desktop/mobile browser acceptance do not transfer to this successor. Historical measurements and all numerical tables are retained; new practical guidance is explicitly interpretation or recommendation, not new inference, a training experiment or production authorization.

## What should a reader take away in practice?

**Interpretation and recommendations, not additional experimental findings.** Decision models are potentially useful for repeated, bounded semantic judgments, but this study has not demonstrated a net benefit for Future Shock's workflow. A typed interface can make an answer easy for software to consume and reject malformed output. It does not guarantee that the answer means the right thing, follows the evidence, or deserves authority over an action.

The distinction matters in ordinary work. A service might read a support message correctly, assign a sensible category and still route it to the wrong team because the menu omits the team that actually owns the problem. A news ranker might put more relevant stories near the top while making poorly calibrated claims about how likely any story is to matter. Reading, rule execution, evidence sufficiency, ranking, calibration and escape behavior therefore need separate tests. Success in one is a reason to investigate that capability, not to assume the others work.

For a low-risk, repeated semantic task, a bounded model is worth considering alongside simpler alternatives. For an exact executable rule, use code. For a decision whose necessary evidence is absent, obtain that evidence or defer. Neither a constrained output nor fine-tuning can make an inadequate decision menu complete by itself. The practical details below distinguish plausible uses, training choices and proposed adoption criteria from the measurements retained in this page.

### Which question do you want to answer?

- [Where might these models be useful, and where are they a poor fit?](#practical-fit)
- [When would training a model make sense?](#training)
- [What would justify using one at Future Shock?](#fs-adoption)
- [What did the experiments actually measure?](#measured-comparison)
- [What did the new Jev prerequisite run establish?](#jev114-prerequisites)

## What does a constrained answer establish?

A decision interface can require a legal answer without ensuring that the answer follows from the supplied evidence. The useful question is which parts of the decision it gets right: reading an observation, applying a rule, recognizing missing information, choosing an available answer, using an escape option, or ranking work for review. These experiments measure those behaviors separately. They do not yield an overall intelligence score.

The record contains substantial successes. Every current arm read the supplied cells correctly in the larger control cohort. Every arm selected the correct answer when it was available in the first menu probe and used the escape when the correct content answer was omitted. Simple rule lookup also worked across the small paired presentations. Those findings matter alongside failures on the larger computation task, distinctions between two escape types, and separate questions about whether the menu contains an answer. [prerequisites] [menu] [small_rules] [exits] [coverage]

### Findings at a glance

- **Reading and rule execution separate.** All current arms passed reading, but none reached the frozen computation gate on the larger one-step task. Perplexity scored above historical Jev on these matched cases. [prerequisites]
- **Ranking and threshold behavior separate.** Clef Flash ranked computational sufficiency well but recognized no determined cases at the fixed threshold. Perplexity recognized more, while also producing false certainty and incorrect target bits. [sufficiency]
- **Escape options worked on easy fixtures, with limits.** The first explicit escape was consistently used correctly. The richer two-exit follow-up exposed errors in Clef and Perplexity on these finite fixtures; it also changed domains, seed count and content alternatives, so it does not isolate an exit-count effect. [menu] [exits]
- **Equivalent questions need not produce equivalent probability judgments.** The coverage diagnostic found differences within the same states; its changed common context prevents attributing improvement over the earlier menu probe to wording alone. [coverage] [context_limit]
- **Better news ranking has not demonstrated saved work.** Perplexity beat the historical Jev ranking at the selected review budget but did not exceed the available historical baselines. Current-workflow ranking and measured reviewer time are unavailable. [utility] [utility_limit]

## What was compared?

The current comparison contains four hosted services. The literal requested identifiers and serving providers below are retained to distinguish a product name from a route. In particular, the Cloudflare namespace in a Clef identifier does not identify the final serving provider. Hosted weights, tokenizer behavior and absence of truncation were not independently established. [models] [methods]

| Arm | Requested canonical ID | Serving provider |
| --- | --- | --- |
| Microsoft Decision 1 | microsoft/microsoft-decision-1-20261009 | Azure |
| Clef Flash | cloudflare/clef-flash | PrimeIntellect |
| Clef | cloudflare/clef | PrimeIntellect |
| Perplexity Decider V1.1 27B | perplexity/pplx-decider-v1.1-27b-20261006 | Perplexity |

**Integration evidence — Canonical requested route and provider metadata are accepted supplemental evidence, not full-export fields.** Public source/protocol citation pending.

Historical Jev 1.13 is a separate response-reuse track. Its matched prerequisite and exploratory scores use the same case identifiers and target semantics; they are not contemporary reruns. The corrected semantic-v3 news run is the relevant historical news baseline. Older news versions with extra heuristic fields or malformed tag arrays are not substituted. Untuned Laya is reported separately below and is neither a fresh fifth arm nor an adapted checkpoint comparison. [prerequisites] [sufficiency] [utility] [laya] [methods]

### Cohorts and sampling units

The larger CA controls compare reading and one-step rule execution on the same assigned cases across services. The exploratory sufficiency branch uses a fixed matched subset after the prerequisite failure. Original symbolic/categorical controls and the new paired mini-fixtures are separate cohorts. Menu, exit and coverage tests use small hand-authored deterministic seed sets, with several related presentations or conditions per seed. News uses historical editorial records and proxy labels. [prerequisites] [sufficiency] [controls] [representation] [menu] [exits] [coverage] [news]

| Cohort (four arms) | Assigned | Recorded outcomes | Valid | Invalid | Uncalled | Source units / dependence |
| --- | --- | --- | --- | --- | --- | --- |
| Reading | 240 | 240 | 240 | 0 | 0 | 60 shared cases |
| One-step | 2116 | 2116 | 2116 | 0 | 0 | 529 shared cases |
| Exploratory sufficiency | 720 | 720 | 719 | 1 | 0 | 180 shared cases; dependent CA families |
| Original domain controls | 128 | 128 | 128 | 0 | 0 | 32 shared fixtures |
| Paired mini-fixtures | 640 | 640 | 639 | 1 | 0 | 40 seed groups; four siblings |
| Menu probe | 288 | 288 | 288 | 0 | 0 | 8 seeds; menus and presentations |
| Two-exit follow-up | 144 | 144 | 144 | 0 | 0 | 12 matched seed groups |
| News calibration + test | 2200 | 2200 | 2199 | 1 | 0 | 550 shared historical rows; possible event clusters |
| Coverage diagnostic | 320 | 320 | 320 | 0 | 0 | 4 seeds; 192 initial + 128 planned repeats |

**Integration evidence — Assigned/valid/invalid regenerated. Recorded-outcome, uncalled and dependence descriptions remain accepted supplemental evidence.** Public source/protocol citation pending.

Recorded outcomes include terminal failures, not necessarily usable provider answers. All assigned outcomes are mapped and no suffix remains uncalled. Unknown billing is a separate accounting property, not a count of scientifically valid responses. [inventory] [accounting]

The cumulative attempt count includes services, siblings, planned repeats and smoke requests. It is not a count of independently sampled problems. The cohorts have been repeatedly inspected and used to design follow-ups; packaging them into a reusable suite creates a historical development/regression release, not a fresh held-out benchmark. [accounting] [pending]

## Can the models read evidence and apply a supplied rule?

The larger prerequisite task supplies an eight-entry local rule table and indexed observations on a circular, width-64 elementary cellular automaton. Reading asks for a cell already supplied. One-step execution requires finding the relevant neighborhood and applying the rule. A deterministic lookup/simulator is the exact-answer baseline. The scoring rule predicts one when P(cell is one) is at least 0.5. [methods] [prerequisites]

| Model / track | Reading correct | One-step correct | One-step accuracy | Current computation gate |
| --- | --- | --- | --- | --- |
| Jev 1.13 (historical) | 60/60 | 377/529 | 71.3% | historical |
| Microsoft Decision 1 | 60/60 | 285/529 | 53.9% | failed |
| Clef Flash | 60/60 | 291/529 | 55.0% | failed |
| Clef | 60/60 | 304/529 | 57.5% | failed |
| Perplexity Decider V1.1 27B | 60/60 | 459/529 | 86.8% | failed |

**Integration evidence — Counts and accuracy regenerated; frozen gate status retained from accepted protocol evidence, not inferred from aggregate accuracy alone.** Public source/protocol citation pending.

All four current arms failed the frozen 95% computation gate despite perfect reading. Perplexity's higher one-step score than historical Jev is a result on this matched cohort, not proof of general superiority. Failing this gate restricts the interpretation of the subsequent computational sufficiency experiment; it does not invalidate unrelated menu or news measurements. [prerequisites] [sufficiency]

### Do smaller controls tell the same story?

Original symbolic and categorical controls were small and uneven in class coverage. The categorical targets omit q2. The accepted gates considered represented-class performance as well as aggregate accuracy: the symbolic gates failed for all current arms, while the categorical gates passed. Passing those controls did not activate the longer suite. [controls] [methods]

| Model | Symbolic reading | Symbolic one-step | Categorical reading | Categorical one-step |
| --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 6/8 | 6/8 | 8/8 | 8/8 |
| Clef Flash | 4/8 | 6/8 | 8/8 | 8/8 |
| Clef | 6/8 | 6/8 | 8/8 | 8/8 |
| Perplexity Decider V1.1 27B | 7/8 | 6/8 | 8/8 | 8/8 |

**Integration evidence — Numerators regenerated; eight assigned per task retained from accepted cohort contract.** Public source/protocol citation pending.

The new mini-fixtures isolate lookup, neighborhood extraction, combined extraction and rule application, multistep execution, and partial-information judgments. They use eight seeds per family, each in four semantically equivalent presentations. [small_rules]

| Model | Lookup | Extraction | Combined | Multistep | Partial |
| --- | --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 32/32 | 29/32 | 32/32 | 18/32 | 19/32 |
| Clef Flash | 32/32 | 18/32 | 31/32 | 20/32 | 20/32 |
| Clef | 32/32 | 26/32 | 32/32 | 19/32 | 21/32 |
| Perplexity Decider V1.1 27B | 32/32 | 32/32 | 32/32 | 20/32 | 25/32 |

**Integration evidence — Numerators regenerated; 32 assigned per task, not valid-only by_task n.** Public source/protocol citation pending.

These results rule out the sweeping claim that the services cannot apply supplied rules. They also show why a task label is insufficient provenance: the easy combined fixtures differ from the larger width-64 task. Different families use different seeds, so their score differences do not isolate the causal effect of computation depth. Flash's extraction result retains its invalid response in the assigned denominator; its valid-only denominator would be smaller. [small_rules]

## Can they identify when the answer is determined?

Here, sufficient information means that every permitted completion of the hidden initial cells produces the same target value. It is an exact property of the task, not a confidence rating. The exploratory slice contains covered-and-determined, uncovered-and-ambiguous, and uncovered-but-determined cases. The latter prevents geometry alone from answering the question. Determined target bits are imbalanced, so simply predicting the more common bit is not evidence of reliable computation. [sufficiency] [methods]

| Model / track | Valid / 180 assigned | Sufficiency AUC | Determined recognized | False certainty | Sufficient + correct bit |
| --- | --- | --- | --- | --- | --- |
| Jev 1.13 (historical) | 180 | 0.812 | 3/120 | 0/60 | 3/120 |
| Microsoft Decision 1 | 180 | 0.861 | 44/120 | 4/60 | 29/120 |
| Clef Flash | 180 | 0.911 | 0/120 | 0/60 | 0/120 |
| Clef | 180 | 0.875 | 5/120 | 0/60 | 3/120 |
| Perplexity Decider V1.1 27B | 179 | 0.904 | 100/120 | 12/60 | 73/120 |

**Integration evidence — All numerical cells regenerated; exploratory-only after failed prerequisite.** Public source/protocol citation pending.

AUC measures ranking on valid probability responses; it does not determine whether a fixed threshold works. Clef Flash's high AUC coexists with zero recognized determined cases at 0.5. Perplexity recognized more determined cases but also asserted sufficiency on ambiguous cases, and recognition did not guarantee a correct bit. The joint endpoint requires both recognition and the correct value. The Perplexity rate-limit failure remains an invalid assigned outcome without an invented probability. [sufficiency]

These are 180-case exploratory results per arm following failed computation prerequisites. The separate historical Jev 3,600-case study is not this table, and a new 3,600-case main study has not run for the current arms. No threshold was retrospectively selected here to turn a poor fixed-threshold result into a success. [sufficiency] [methods]

## What happens when the correct answer is missing from the menu?

The first menu probe used eight deterministic seeds in two domains. Each seed appeared with a complete menu plus escape, the correct answer omitted plus escape, and the correct answer omitted without escape. Base, reversed-order and explicitly mapped opaque-label presentations yielded 24 assigned presentations per condition per arm. All 288 responses were valid. [menu]

| Model | Full-menu correct | Omitted + escape correct | Forced confidence >=.9 | Coverage full | Coverage omitted + escape | Coverage forced |
| --- | --- | --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 24/24 | 24/24 | 0/24 | 17/24 | 5/24 | 9/24 |
| Clef Flash | 24/24 | 24/24 | 0/24 | 24/24 | 13/24 | 24/24 |
| Clef | 24/24 | 24/24 | 0/24 | 24/24 | 22/24 | 24/24 |
| Perplexity Decider V1.1 27B | 24/24 | 24/24 | 0/24 | 15/24 | 9/24 | 9/24 |

**Integration evidence — Semantic and coverage counts regenerated. Forced confidence >=.9 is absent from full export: retained only as accepted supplemental source evidence, not replay-verified here.** Public source/protocol citation pending.

The available-answer and escape decisions were perfect across these presentations. The forced selections did not reach the predeclared 0.9 threshold in either reported confidence or chosen-option probability. This qualifies the historical concern about confidently selecting a false menu option: the interface still cannot express an answer it does not offer, but these fixtures did not produce high-confidence forced errors. Ordinary semantic accuracy is undefined in the forced condition because every legal content answer is false by construction. [menu]

The coverage question asks whether a correct content category is listed, explicitly excluding the escape. Its answers were less reliable than the choices themselves. A model can therefore make the desired selection without reliably answering this separate proposition about the menu. That observation does not identify an internal reasoning mechanism. [menu]

### Can it distinguish known absence from insufficient evidence?

The follow-up supplied two distinct exits and plausible content alternatives. Twelve matched seed groups each appeared with the answer present, a known answer absent, or evidence genuinely insufficient. All 144 responses were valid. [exits]

| Model | Answer present | Known absent: correct exit | Insufficient: correct exit | Absent: content errors | Absent: exit confusions |
| --- | --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 12/12 | 12/12 | 12/12 | 0 | 0 |
| Clef Flash | 12/12 | 12/12 | 12/12 | 0 | 0 |
| Clef | 12/12 | 4/12 | 12/12 | 3.0 | 5.0 |
| Perplexity Decider V1.1 27B | 12/12 | 9/12 | 12/12 | 0 | 3.0 |

**Integration evidence — All numerical cells regenerated from condition groups.** Public source/protocol citation pending.

Every arm identified insufficient evidence correctly on these fixtures. Microsoft and Flash also handled known absence correctly; Clef sometimes chose an inappropriate content answer or confused the exits, while Perplexity's known-absence errors were exit confusions. None of the errors reached the frozen high-confidence or selected-probability threshold. The earlier perfect single-escape result therefore does not transfer uniformly to this richer menu. These finite lookup tasks should not be blended with difficult computational determinacy into a single uncertainty score. [exits]

## How sensitive are decisions to presentation and question structure?

### Equivalent representations

The paired rule diagnostic mapped choices and probability keys back into a common semantic space before comparing the base, complement, symbol rename and combined transformations. Complementing changed the rule and target as well as the input. Stable wrong answers count as consistency, not correctness. [representation] [cluster_methods]

| Model | Complete / assigned seed groups | All four correct | Same semantic choice | Consistent but wrong |
| --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 40/40 | 27 | 30 | 3 |
| Clef Flash | 39/40 | 24 | 27 | 3 |
| Clef | 40/40 | 26 | 30 | 4 |
| Perplexity Decider V1.1 27B | 40/40 | 32 | 34 | 2 |

**Integration evidence — Complete-group counts regenerated from exported semantic pairs; no incomplete-group imputation.** Public source/protocol citation pending.

Flash has one incomplete seed group because its base extraction response failed normalization. Its other siblings remain in per-case accuracy, but no base comparison is imputed. This cohort had no identical-packet repeat controls, so observed differences cannot be apportioned between representation effects and provider or sampling variation. [representation] [cluster_methods]

### Coverage wording, bundled questions and planned repeats

The completed diagnostic selected four seeds by an outcome-independent rule. It crossed two coverage states, three wordings and two call structures for each service. Negative wording was reverse-scored. The initial allocation contains 192 requests; 128 additional, predeclared identical-wire repeats apply only to the original wording. All 320 responses were valid. [coverage] [context_limit]

| Model | Initial coverage correct | Complete repeat triplets | Mean signed-p spread | Max signed-p spread |
| --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 46/48 | 16 | 0.0106968 | 0.0532620 |
| Clef Flash | 48/48 | 16 | 0.0000625 | 0.0010000 |
| Clef | 34/48 | 16 | 0.0000000 | 0.0000000 |
| Perplexity Decider V1.1 27B | 48/48 | 16 | 0.0000000 | 0.0000000 |

**Integration evidence — All numerical cells regenerated from initial cells and scheduled repeat spreads.** Public source/protocol citation pending.

All bundled decision choices were correct across wordings and repeats. Clef's omitted-answer states illustrate why coverage needs its own endpoint: original wording was correct in four of four bundled cases but only one of four standalone cases; reverse-scored negative wording was correct in one bundled case and none standalone. Its original-wording repeated probabilities were identical in all recorded triplets. Microsoft made two initial coverage errors, both with negative wording on omitted-answer states. [coverage]

The full menu and selection rules were moved into common state so that bundled and standalone calls had identical evidence. That is a context change from the first menu probe, and the diagnostic also uses a smaller preselected seed set. Improved scores cannot establish that wording alone fixed earlier errors. Repeated rounds were serial: unchanged probabilities in these observations do not prove general determinism, while observed spreads cannot separate period drift from sampling variability. [context_limit] [coverage]

## Does improved news ranking reduce review work?

The applied track uses 550 historical rows: the earliest 330 for calibration and the later 220 for test. Test labels contain 79 proxy-important and 141 proxy-negative stories. Historical pass/qualify labels map to important; hold/kill map to not important. Those labels came from an editorial/LLM chain and have not become independent truth through reuse. Semantic packets exclude URLs, heuristic scores and post-adjudication labels. Calibration choices were frozen before test scoring; all current-arm test responses were valid. [news] [methods]

| Model | Valid test | Accuracy at .5 | AUC | TP | FP | FN | Brier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 220 | 53.6% | 0.660 | 68 | 91 | 11 | 0.364 |
| Clef Flash | 220 | 48.6% | 0.693 | 76 | 110 | 3 | 0.345 |
| Clef | 220 | 53.2% | 0.700 | 74 | 98 | 5 | 0.309 |
| Perplexity Decider V1.1 27B | 220 | 56.4% | 0.732 | 74 | 91 | 5 | 0.324 |

**Integration evidence — Valid/accuracy/AUC/Brier regenerated. TP/FP/FN absent from full export: retained only as accepted supplemental comparison evidence.** Public source/protocol citation pending.

The default threshold overpredicts importance. Ranking can improve even when threshold accuracy remains poor, so neither default accuracy nor the majority-class baseline answers the review-allocation question. Calibration-selected routing likewise has to be assessed for disagreement, not only for how few important stories it misses. [news] [routing]

| Model | Frozen threshold | Autonomous /220 | Fallback /220 | Routed disagreement | Important miss rate |
| --- | --- | --- | --- | --- | --- |
| Microsoft Decision 1 | 0.8 | 157 | 63 | 45.2% | 1.3% |
| Clef Flash | 0.7 | 180 | 40 | 49.4% | 0.0% |
| Clef | 0.75 | 142 | 78 | 41.5% | 0.0% |
| Perplexity Decider V1.1 27B | 0.9 | 121 | 99 | 36.4% | 0.0% |

**Integration evidence — All numerical cells regenerated. Current-arm research policy scores; not production authorization.** Public source/protocol citation pending.

For a valid importance probability `p` and frozen threshold `t`, route positive when `p >= t`, negative when `p < t` and `p <= 1-t`, and otherwise to fallback. At `t = 0.5`, the positive route wins the shared boundary. Search the calibration-only grid `0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95, 1.00` for maximum autonomous coverage subject to an important-miss rate no greater than 2.5%, breaking coverage ties toward the lower threshold. The historical selector computes coverage and important-miss rate on valid calibration rows; important misses are proxy-positive rows routed negative, divided by valid proxy-positive rows. Missing or invalid responses always fall back without probability imputation; if no threshold qualifies, no autonomous route is allowed. The saved thresholds are 0.8/0.7/0.75/0.9 in table order and were frozen before test dispatch. All test rows here are valid, so the displayed test important-miss denominator is all 79 proxy-positive stories. [routing] [methods]

These are research policy scores, not deployed autonomous judgments. The high disagreement rates are incompatible with interpreting the low important-miss rates as dependable autonomous editorial selection. [routing]

### Equal review budget

The offline utility analysis selected a 50% review budget using calibration and then compared 110 review slots on the same test split. [utility]

| Ranking | Review slots | Proxy-important caught | Proxy-negative slots | Recall |
| --- | --- | --- | --- | --- |
| historical_significance_path | 110 | 64/79 | 46 | 81.0% |
| jev_semantic_v3 | 110 | 49/79 | 61 | 62.0% |
| perplexity | 110 | 56/79 | 54 | 70.9% |
| stored_phase_a_heuristic | 110 | 57/79 | 53 | 72.2% |

**Integration evidence — Review-budget endpoints absent from both integrated exports. Entire table retained as accepted supplemental practical-shadow evidence, not full-suite replay parity; no measured time saved.** Public source/protocol citation pending.

The historical significance path reads `Number(significance ?? heuristic_significance ?? 1) / 10`; the stored Phase-A baseline reads `heuristic_significance / 10`. The former was once mislabeled “heuristic alone.” Its stored information may include later changes or information unavailable in the semantic packet. It is an operational historical comparator, not an equal-information causal control. [utility_limit]

Perplexity retrieved more proxy-important items than historical Jev at this budget but did not demonstrate an advantage over these historical baselines. Its one-story gap from Phase-A is descriptive, not evidence of statistical superiority for the heuristic. The actual current workflow ranking and measured reviewer minutes remain unavailable. Cheap or fast component inference does not measure retrieval, disagreement resolution, fallback, correction or review time. No prospective human-news workflow trial has completed. [utility] [utility_limit] [methods]

## Historical untuned Laya: a separate track

The saved untuned cross-domain diagnostic reports failed competence controls. Its longer tasks exist, but corresponding longer runs have not been completed for the four current arms. Better current control results cannot stand in for missing long-horizon comparisons. No inference about a trained Laya checkpoint follows from this table. [laya]

| Historical untuned Laya task | n | Accuracy |
| --- | --- | --- |
| symbolic/full_execution | 400 | 50.0% |
| symbolic/partial_sufficiency | 200 | 42.5% |
| categorical/full_execution | 300 | 62.3% |
| categorical/partial_reachable_set | 200 | 50.0% |
| symbolic/reading_control | 8 | 50.0% |
| symbolic/one_step_control | 8 | 50.0% |
| categorical/reading_control | 8 | 37.5% |
| categorical/one_step_control | 8 | 25.0% |

**Integration evidence — Historical summary-only values regenerated; no case parity or fresh fifth arm.** Public source/protocol citation pending.

## Where might decision models be useful?

**Plausible applications, not validated deployments from this study.** The strongest practical case is a stream of similar decisions whose meanings can be stated clearly, whose errors can be detected, and whose consequences are reversible. A bounded semantic model could suggest the destination of a support ticket, assign a taxonomy label, order a review queue, suggest an appropriate tool, flag possible duplicates, or recommend escalation to a person. Those applications are candidates for evaluation, not a list of tasks this comparison proved safe or economical.

The attraction is the interface. Software can receive a named category or score without extracting it from a paragraph, then apply explicit fallback rules. That can make integration simpler, but the useful unit is the whole workflow. A duplicate flag should invite comparison before records are merged; a tool recommendation should remain subject to permission and input checks; an escalation score should not remove the route to a human. A confident legal category can still be wrong, and a probability over the offered choices cannot certify that those choices cover the situation.

The poor fits follow from those limits. Exact rules that can be executed reliably in code do not need a learned approximation. Missing evidence calls for retrieval or abstention, not a more decisive score. Bad or overlapping menus need redesign and appropriate escape options. Irreversible or costly decisions need protections beyond a bounded answer, while low-volume tasks may never recover the cost of labeling, evaluation and maintenance. Even a useful model can be the wrong engineering choice when a heuristic or ordinary classifier does the job more cheaply and transparently.

The experimental record supports keeping these possibilities open without claiming they have already paid off. Perfect reading and the easy menu successes demonstrate bounded capabilities on these fixtures. The harder rule failures and the distinction between recognizing uncertainty and getting the answer right constrain what can be inferred from them. None of those results alone establishes a business case, or makes semantic classification generally useless.

## When would training a model make sense?

**Engineering guidance, not a measured training result.** Training for an organization's decisions need not mean building a foundation model from scratch. There are three practical routes worth comparing, with the least complicated adequate system as the starting point.

- **A conventional classifier or embeddings-based baseline.** If the categories are stable and labeled examples are available, a small classifier or a classifier over text embeddings provides a useful reference. It tests whether a more elaborate decision model is necessary at all.
- **Fine-tuning an existing open-weight model.** Adaptation can be investigated when general-purpose behavior misses stable, organization-specific meanings. The relevant experiment is whether a tuned candidate improves the protected evaluation relative to its untuned version and credible alternatives, after accounting for operating costs. Improvement is a hypothesis to test, not a consequence of having run training.
- **Distillation from independently audited labels.** A stronger model can propose labels or candidate explanations, but those outputs should not become unquestioned truth. Independently review label quality, preserve disagreements and hard negatives, and train a smaller system only on a dataset whose provenance and errors are understood. A model trained to reproduce an unaudited teacher can reproduce its mistakes more efficiently.

Cloudflare's primary announcement says that Clef weights are released under Apache 2.0 and describes fine-tuning services beginning with a hands-on forward-deployed engineering team, followed later by a self-serve platform.[[1]](https://blog.cloudflare.com/clef-decision-models) This establishes an announced adaptation route; it does not establish general availability of the self-serve platform, customer eligibility, or measured fine-tuning benefit in this study. Jev fine-tunability is not established by the evidence assembled here, so this guidance does not assume it. Historical untuned Laya is not a trained-versus-untuned comparison and cannot estimate what a trained checkpoint would gain.

Training is worth considering when the decision meaning is stable enough to learn, sufficiently specific to the organization to justify adaptation, and frequent enough for a benefit to exceed labeling and maintenance costs. The dataset needs quality labels, negative examples and edge cases, including ambiguous cases and cases that should fall back. Reserve a protected chronological, event-grouped holdout before tuning: related accounts of one event must not leak across training and evaluation. Keep that holdout out of prompt selection, threshold selection and repeated training decisions; later iterations need fresh evaluation windows.

Compare the candidate with a simple classifier, a heuristic and a prompted general model, as well as the untuned model and the actual workflow it would replace. Measure correction effort, monitoring, drift and retraining costs rather than treating inference cost as the whole saving. A failed cellular-automaton prerequisite does not by itself create a need to train. If the intended job is an exact rule, code remains the appropriate baseline; if the intended job is semantic triage, evaluate that job directly. No fine-tuning experiment or measured training gain is reported here.

## What would justify using one at Future Shock?

**Current recommendation, not a production decision.** The immediate demonstrated use of this work at Future Shock is research and a reusable development/regression suite. A separate prospective production trial has not been committed to, and this page grants no authority to deploy one. The potentially useful next application is advisory ranking in a shadow test, where the model's ordering can be compared with the real review queue without discarding stories or changing publication decisions.

The historical news comparison explains why the distinction is necessary. At 110 review slots, the stored significance path caught 64 proxy-important stories, the stored Phase-A heuristic 57, Perplexity 56 and historical Jev 49. These are the accepted historical review-budget endpoints, not fresh measurements. The significance path had potentially unequal information, and all four counts use proxy labels rather than fresh independent editorial judgments. The actual current workflow ranking and workflow timing were not measured. Perplexity's improvement over Jev is therefore not evidence that replacing Future Shock's workflow would save work or improve coverage.

An advisory ranker would suggest what a reviewer reads first while leaving all items available. It would not decide whether evidence is sufficient, whether a claim is factual, or whether an article should be published. Preserve the separation between creator and validator: a model that proposes a story, label or explanation should not be its own sole source of acceptance. Independent source checking and editorial judgment remain necessary even when a score is high. No single score warrants factual or publication authority.

### Proposed acceptance criteria for a separate trial

**NEW RECOMMENDATIONS. These are proposed targets, not frozen experiment gates, achieved findings, universal standards or user authorization to deploy.** They are deliberately different for different losses: an inconvenient ordering is reversible, a wrong tag creates cleanup, and an automatically discarded important story can disappear from review altogether. Set the deployment loss assumptions and review the targets before a trial; adjust them to those consequences rather than presenting these numbers as natural constants. The historical 95% CA computation gate remains unchanged and separate.

**Advisory ranking.** Use fresh, independently labeled material with a chronological, event-grouped evaluation design. Compare against the actual current workflow at equal review effort. Require either better important-story recall at that effort, or matched recall with at least a 20% measured reduction in review time, including correction and maintenance effort. The 20% proposal asks for enough margin to justify integration and continuing upkeep; it is not an observed saving. Repeat the comparison across time windows, report uncertainty, and inspect topic and small-source subgroups so that an aggregate gain does not conceal poorer coverage of less prominent sources. Keep items accessible throughout; ranking need not introduce automatic discard.

**Low-risk tagging.** For any category proposed for automatic assignment, aim for at least 95% precision on the automatically handled subset, with meaningful coverage measured per category and fallback for everything outside that subset. Precision here means the share of automatic labels that independent review judges correct; coverage means the share of eligible items the system actually handles automatically. Specify useful coverage and uncertainty requirements before evaluation, rather than achieving precision by labeling almost nothing or hiding a weak category in an aggregate. The proposed precision target limits routine cleanup for reversible labels; it is not the historical CA accuracy gate and has not been achieved for an FS tagging deployment in this study.

**Automatic rejection, if ever separately justified.** Require an important-story miss rate below 1%, supported by an independently labeled sample and uncertainty analysis. Define the denominator as all independently judged important stories in the evaluated stream, and the numerator as those routed to automatic rejection. The stricter proposal reflects the cost of losing an item before anyone reads it. Zero observed misses in a small sample is not proof of safety; sample size, event dependence and uncertainty must support the claim. There is no need to add automatic discard merely because a ranker is useful, and this page does not recommend deploying it now.

**Factual validation and publication.** None of those targets would authorize a model to validate its own evidence or publish an article. Those responsibilities require independent validation and an explicitly authorized editorial process, not a higher confidence threshold. A favorable shadow result would justify reviewing the next bounded experiment, not quietly widening the model's authority.

The practical conclusion is conditional but useful: repeated bounded semantic judgments remain a plausible role for these models, and Future Shock has no demonstrated net workflow benefit yet. A carefully scoped comparison could resolve that question without treating every model failure as a reason to abandon the category, or every cleanly typed answer as a reason to trust it.

The linked Cloudflare source is a vendor primary announcement, used only for the Apache 2.0 weights and announced fine-tuning route. Its benchmark, speed and training-benefit claims are not adopted as findings of this study. The applications, training choices and deployment targets above are recommendations; the historical measurements retain their original evidence boundaries below and in the preceding results.

## Sources

[1] https://blog.cloudflare.com/clef-decision-models — Introducing Clef: our open-source decision models, and new RL fine-tuning platform
## Accounting and evidence status

| Accounting quantity | Value |
| --- | --- |
| attempts | 6798 |
| charged_input_tokens | 32035672 |
| cap_charge_usd | 0.970399942 |
| estimated_usd | 0.970399942 |
| billed_known_usd | 0.140790438 |
| pending | 0 |
| unknown_usage | 3 |

**Integration evidence — Billing/token/reserve quantities absent from full export. Accepted historical snapshot retained, not a current bill or new replay measurement. 6798 reserves = 6796 task instances + two smoke attempts.** Public source/protocol citation pending.

Known reported billing is only the known portion of the bill. Unknown usage/billing remains unknown, and conservative cap accounting is a separate quantity. The cumulative total includes the two smoke attempts and does not count inherited pilot responses twice. It does not aggregate all historical Jev or Laya work into an all-time total. The historical reserve snapshot reconciles to 6,796 verified task instances plus two smoke attempts. Private accounting evidence is not distributed. [accounting]

## Methods appendix

### Response types, scores and thresholds

CHOICE supplies a probability distribution over the offered menu; NOUL supplies a probability-like score for an absolute proposition. Reported confidence, selected-option probability, absolute proposition probability and token likelihood are distinct measurements. Binary bit and sufficiency predictions use `p >= 0.5`. Coverage reverses negative-wording probability as `1 - raw` before applying that threshold. The coverage false-certainty rule is signed `p >= 0.9` for absent answers or `p <= 0.1` for present answers. Menu and exit false-high-confidence checks use the predeclared 0.9 threshold separately for reported confidence and selected-option probability. [methods] [menu] [exits] [coverage]

AUC reports ranking, with tied positive/negative pairs receiving half credit in the historical recomputation. Brier error is squared probability error against the binary reference. Accuracy and joint correctness retain assigned denominators for demonstrated successes; probability metrics exclude invalid observations and state the smaller valid denominator. AUC with only one truth class and forced-menu semantic accuracy are undefined, not zero. News references are editorial proxies; deterministic task oracles and editorial labels are not interchangeable. [sufficiency] [news] [methods]

### Amendments, errors and no retry

A documented prospective successor accepted any exact co-maximum option rather than requiring the first declared maximum. The original stop and settlement were retained, and the accepted policy was applied through a hash-bound overlay; near ties were not made exact. Clef Flash's distribution summing to 1.0003 remains invalid against the frozen 0.0002 normalization tolerance. It was not renormalized. Two Perplexity rate-limit failures remain failures. [methods]

Failed requests were not retried. Approved continuations inherited authenticated prior outcomes and called only untouched assignments. Planned coverage repeats are a different scientific object: each has a distinct trial identity despite identical wire bytes and was fixed before outcomes were observed. No best-of-N selection or cache collapse is appropriate. Oracle information belongs in scoring, not the actor packet. [methods] [coverage]

### Dependence and uncertainty

The paired-analysis intervals use 95% percentile cluster bootstrapping with 3,000 draws, fixed seed 20261014 and stable per-table streams. Whole seed groups are resampled, retaining sibling observations and invalidity patterns. Pooled representation rows use 40 seed groups and family rows eight; the coverage diagnostic has only four source seeds. These are descriptive intervals for a small development diagnostic, with many exploratory contrasts, not confirmatory tests. Boundary intervals do not prove certainty. Full saved interval fields are available in the internal paired-analysis source rather than selectively promoted in the main table. [cluster_methods]

News stories can describe the same event, so chronology alone does not establish event independence. This page supplies no new significance test or unmeasured paired uncertainty for the news-budget comparison. Follow-up designs were informed by earlier observations, and packaging the record does not make those design choices retrospectively preregistered. [news] [utility] [pending]

## Reproducibility and release boundaries

The historical offline suite is accepted, as are the predecessor historical reconstruction, sanitized aggregate integration and accepted Jev prerequisite page with its desktop/mobile HTML visual QA. Those receipts do not accept this new page. Independent methods/claims review and browser QA of this successor remain pending. The unchanged historical four-arm inventory is not expanded by the new prerequisite attempts.

Historical Jev news calibration remains qualifies=false and ineligible for autonomous routing. Laya remains summary-only. Other Jev 1.14 families are unrun/unavailable, not zero-filled or borrowed from historical Jev.

Stable relative download links are provided for this local bundle. Public historical source/protocol citations, distribution rights/privacy clearance and external publication authorization remain pending. Bracketed historical claim identifiers are internal labels, not public citations. Nothing was published. The aggregate downloads do not constitute a self-contained raw-replay kit.

Local reproduction: run python3 research-release-practical/verify.py from the private study checkout. See METHODOLOGY.md and DATA-DICTIONARY.md.

## Historical Jev fixed-threshold news (separate track)

| Split | Assigned | Valid | TP | FP | TN | FN | AUC |
| --- | --- | --- | --- | --- | --- | --- | --- |
| news_calibration | 330 | 330 | 108 | 50 | 125 | 47 | 0.738083 |
| news_test | 220 | 220 | 49 | 62 | 79 | 30 | 0.648442 |

Source/protocol public citation pending. Historical provider/time/renderer confounds remain; no controlled causal superiority claim.

## Jev 1.14 audited prerequisite addendum

New native TypeSafe Jev 1.14: reading 60/60 (PASS); one-step 356/529, 527 valid and 2 invalid (FAIL). The frozen gate requires all responses valid, at least 95% overall and at least 95% in each class. No automatic downstream promotion follows.

| Track / stage | Assigned | Valid | Invalid | Correct | Assigned accuracy | Valid-only accuracy | Gate |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Jev 1.14 native / reading | 60 | 60 | 0 | 60 | 100.0000% | 100.0000% | PASS |
| Jev 1.14 native / one_step | 529 | 527 | 2 | 356 | 67.2968% | 67.5522% | FAIL |
| Jev 1.13 historical / reading | 60 | 60 | 0 | 60 | 100.0000% | 100.0000% | PASS |
| Jev 1.13 historical / one_step | 529 | 529 | 0 | 377 | 71.2665% | 71.2665% | FAIL |

### Historical matched outcomes, not a version experiment

Historical development cases are not fresh held-out trials. Renderer, provider and time are confounded. This descriptive comparison does not establish a causal version effect or general regression. Matched logical cases do not imply identical HTTP bytes or tokenization. Class-specific outcomes shift toward more positive predictions. No significance or population-generalization claim is made.

| Subset | Assigned pairs | Both correct | Only 1.14 correct | Only historical 1.13 correct | Neither correct |
| --- | --- | --- | --- | --- | --- |
| All assigned | 529 | 297 | 59 | 80 | 93 |
| Both valid | 527 | 297 | 59 | 79 | 92 |
| Truth 0, all assigned | 265 | 110 | 6 | 74 | 75 |
| Truth 0, both valid | 265 | 110 | 6 | 74 | 75 |
| Truth 1, all assigned | 264 | 187 | 53 | 6 | 18 |
| Truth 1, both valid | 262 | 187 | 53 | 5 | 17 |

### Invalids and observed usage

One HTTP 529 failure has unknown usage, not zero tokens. One HTTP 200 distribution summed to 0.99 and remains invalid under the frozen 0.0002 tolerance; its known usage is included. Both invalids remain assigned and incorrect; neither supplies an imputed probability. No retry, salvage, replacement or tolerance adjustment is implied.

| Stage | Attempts | Usage known | Usage missing | Known input tokens | Known output tokens |
| --- | --- | --- | --- | --- | --- |
| reading | 60 | 60 | 0 | 50136 | 2520 |
| one_step | 529 | 528 | 1 | 589200 | 47512 |

Observed subtotals: 639,336 input and 50,032 output tokens across 588 usage-known responses; 1 response has unknown usage. These exclude the earlier separate compatibility probe and are not complete token totals or a bill.

Phase-specific timing is available in the aggregate JSON, not pooled into a controlled speed comparison. Timers bracket HTTP transport through raw-event construction, before durable journaling; startup waits and between-call pacing are excluded. Pacing differed across phases. These are not provider compute times, end-to-end throughput or latency guarantees.

| Accounting quantity (not billing) | USD |
| --- | --- |
| Local allowance | 59.970399942 |
| Approved cap | 60 |
| Remaining allowance | 0.029600058 |
| Next-attempt allowance | 0.10 |

Remaining local allowance $0.029600058 is insufficient for the next $0.10 attempt. This is conservative risk-allowance accounting, not verified pricing, spending or an invoice guarantee; billing is unknown. No extra inference or cap change is authorized.

Only the 589 new reading and one-step attempts are complete; they are not additional distinct benchmark cases. Jev 1.14 sufficiency, news, domain/representation, menu/escape, exit and coverage remain unrun/unavailable with null metrics. This closes local page integration only, not the full Jev suite or publication.