When a Decision Model Gives the Wrong Answer

A decision model can return a named category or a score in a format that software can use directly. This is what a typed interface provides: rules about the shape of the answer and the values it may contain. Those rules can reject malformed output, but they cannot establish that a permitted answer follows from the evidence. Repeated tasks that require interpreting meaning may be promising uses; this study has not demonstrated a net benefit for Future Shock's workflow. A separate comparison of current JavaScript scoring with Jev 1.14 now supports retaining the rules for news ranking; its historical labels still do not establish a net workflow benefit. The practical guidance is interpretation and recommendation.

A service might read a support message correctly and still route it to the wrong team because the menu omits the team that owns the problem. A news ranker might put more relevant stories near the top while overstating how likely any particular story is to matter. Those are different failures. We therefore test reading supplied values, executing rules, recognizing whether the evidence determines an answer, ranking cases, and using an escape option separately. A model that succeeds at one has not thereby passed the others.

For a low-risk task repeated often enough to justify evaluation, compare a decision model with simpler alternatives. Use code for an exact executable rule. If the necessary evidence is absent, obtain it or defer the decision. Restricting the output, or fine-tuning the model, cannot by itself repair a menu that leaves out the right answer. The applications, training choices and proposed adoption criteria below follow from these distinctions; they are not measured workflow gains.

Which question do you want to answer?

- Where might these models be useful, and where are they a poor fit?

- When would training a model make sense?

- What would justify using one at Future Shock?

- What did the experiments actually measure?

- What did the new Jev prerequisite run establish?

An answer can satisfy the interface and still be wrong. The tests below ask where that happens: when a model reads a supplied observation, applies a rule, judges whether information is missing, chooses from a menu, uses an escape option, or ranks items for review. An escape option lets the model decline the offered content choices for a specified reason. Keeping these tests separate makes the result more useful than an overall intelligence score.

The services succeeded on several bounded tasks. Each reproduced every supplied cell value correctly in the larger reading control. In the first menu test, each chose the correct answer when it was offered and used the escape when that answer was omitted. Simple rule lookup also worked across the small paired presentations. The harder computation task, the distinction between two reasons to exit, and the separate question of whether a menu contains the answer exposed different limits. Reading and rule tests · Menu tests · Small-rule tests · Two-exit tests · Menu-coverage tests

Findings at a glance

Reading supplied values did not ensure correct computation. All current services passed the reading control, but none met the predeclared computation requirement on the larger one-step task. Perplexity scored above historical Jev on these matched cases. Reading and rule tests

A useful ordering did not ensure a useful decision threshold. Clef Flash ranked cases well by whether their evidence determined an answer, yet recognized none of the determined cases at the fixed threshold. Perplexity recognized more, while also claiming certainty on ambiguous cases and returning incorrect target bits. Determined-answer tests

An escape worked on the first easy tests, but two different exits were harder to distinguish. Clef and Perplexity made errors in the follow-up. That test also changed the domains, number of underlying examples and content alternatives, so it cannot isolate the effect of adding an exit. Menu tests · Two-exit tests

Equivalent questions sometimes received different probability judgments. The menu-coverage test compared questions about the same states. It also changed the context shared by the questions, so improvements over the earlier menu test cannot be attributed to wording alone. Menu-coverage tests · Shared-context limits

Better news ranking has not demonstrated saved work. Perplexity beat the historical Jev ranking at the selected review budget but did not exceed the available historical baselines. The separate current-code comparison below favors the existing JavaScript rules over Jev 1.14 on historical inputs. Measured reviewer time remains unavailable. Equal-budget review · Review-baseline limits

This briefing represents six models with unequal evaluation coverage: four hosted main-suite arms, historical Jev 1.13 comparisons from preserved responses, and a fresh but partial native Jev 1.14 run. A route combines the requested model identifier with the provider that served it. The exact identifiers are shown because a product name alone does not establish that route: Clef's Cloudflare namespace, for example, does not mean Cloudflare served these requests. We did not independently establish the hosted weights, tokenizer behavior, or whether inputs were truncated. Services and routes · Methods

Original-suite inventory, preserved below. The separate October 17 news-ranking comparison adds 169 valid native Jev 1.14 responses against current JavaScript scoring; it does not complete the original news suite or the other unrun families. Current-code comparison

Evidence available: The requested routes and serving providers are documented in retained supporting records, not in the full export. Public source and protocol citations are still pending.

Historical Jev 1.13 results reuse earlier responses to the same identified cases and target meanings; they are not reruns alongside the current services. For news, the comparison uses the corrected semantic-v3 run, rather than older versions with extra heuristic fields or malformed tag arrays. Within the original suite, Jev 1.14 contributes fresh native reading and one-step results only; the version comparison is not contemporaneous and does not isolate a causal version effect. The untuned Laya results below remain a separate historical summary-only track, not a fresh comparable model or a trained-checkpoint comparison. The four-arm cohort counts below describe the main suite, not all six models, and do not incorporate the separate Jev addendum. Reading and rule tests · Determined-answer tests · Equal-budget review · Historical Laya tests · Methods

Which cases were tested?

Each cohort is a set of test cases. The larger cellular-automaton controls, abbreviated CA, ask the services to read supplied values and execute a one-step rule on the same cases. After the computation prerequisite failed, a fixed matched subset was used to explore whether the services could identify sufficient information. The original symbolic and categorical controls and the newer small paired tests are separate cohorts. Menu, exit and coverage tests use small hand-authored sets of underlying examples, or seeds, with related variants of each. The news tests use historical editorial records and labels that stand in for independently established importance. Reading and rule tests · Determined-answer tests · Symbolic and categorical controls · Equivalent-presentation tests · Menu tests · Two-exit tests · Menu-coverage tests · News classification

Evidence available: Assigned, valid and invalid counts were recomputed from the retained data. Recorded-outcome counts, uncalled counts and descriptions of dependence come from supporting records. Public source and protocol citations are still pending.

An assigned case is one the study planned to call; a valid response is one that met its response contract. Recorded outcomes also include failed requests, so the number recorded is not necessarily the number of usable answers. Every assignment here has a recorded outcome and none remains uncalled. Unknown billing is recorded separately and does not determine whether an answer is scientifically valid. Case inventory · Cost records

The attempt total counts calls across services, related variants, planned repeats and small compatibility checks called smoke requests. It does not count independent problems. These cases have already been inspected and used to design follow-ups. They can support development and regression checks, but repackaging them cannot make them a fresh held-out benchmark. Cost records · Reproduction and release limits

The larger prerequisite task uses a cellular automaton: a circular row of 64 cells whose next values follow an eight-entry local rule table. The prompt supplies that table and indexed observations. The reading test asks for a value already supplied. The one-step test asks the model to find the relevant neighboring cells and apply the rule to obtain the next value. Code that looks up the rule and simulates the update supplies the exact-answer baseline. For scoring, the model predicts one when its probability that the cell is one is at least 0.5. Methods · Reading and rule tests

Reading a value is not executing a rule

Reading supplied values

Executing one supplied step

Evidence available: Counts and accuracy were recomputed from the retained data. Gate outcomes come from the accepted protocol records; overall accuracy alone does not determine a pass. Public source and protocol citations are still pending.

All four current arms failed the frozen 95% computation gate despite perfect reading. Perplexity's higher one-step score than historical Jev is a result on this matched cohort, not proof of general superiority. Failing this gate restricts the interpretation of the subsequent computational sufficiency experiment; it does not invalidate unrelated menu or news measurements. Reading and rule tests · Determined-answer tests

Do smaller controls tell the same story?

Original symbolic and categorical controls were small and uneven in class coverage. The categorical targets omit q2. The accepted gates considered represented-class performance as well as aggregate accuracy: the symbolic gates failed for all current arms, while the categorical gates passed. Passing those controls did not activate the longer suite. Symbolic and categorical controls · Methods

Evidence available: Correct-answer counts were recomputed from the retained data. The accepted test specification assigns eight cases to each task. Public source and protocol citations are still pending.

The smaller tests separate looking up a rule, extracting the neighboring cells, combining those steps, executing several steps, and judging partial information. Each task family uses eight underlying examples, each presented in four ways that preserve its meaning. Small-rule tests

Evidence available: Correct-answer counts were recomputed from the retained data. Each task has 32 assigned cases, including invalid responses; the denominator is not restricted to valid responses. Public source and protocol citations are still pending.

The services can apply supplied rules on these easier tests. That does not contradict their failures on the larger width-64 task: the combined small tests and the larger task are different problems. Task families also use different underlying examples, so their score differences cannot isolate the effect of adding computation steps. Flash's extraction score counts its invalid response in the assigned denominator; a score based only on valid responses would use a smaller denominator. Small-rule tests

Information is sufficient here when every permitted way of filling in the hidden initial cells leads to the same target value. The answer is then determined, regardless of how confident the model feels. Some cases reveal all the relevant cells; others hide cells and permit different answers. A third group hides cells but still determines the answer, so the model cannot solve the test just by checking which positions are visible. Among determined cases, one target bit is more common than the other. Repeatedly choosing that bit is not evidence of reliable computation. Determined-answer tests · Methods

Evidence available: All numerical cells were recomputed from the retained data. These results remain exploratory because the computation prerequisite failed. Public source and protocol citations are still pending.

AUC measures how well a score ranks determined cases above ambiguous ones, using valid probability responses. It does not say whether those scores cross the chosen decision threshold. Clef Flash ranked well yet recognized no determined cases at 0.5. Perplexity recognized more, but also called some ambiguous cases sufficient and sometimes returned the wrong bit after recognizing a determined case. The last column requires both recognition and the correct bit. Its rate-limit failure remains an invalid assigned outcome; no probability was invented for it. Determined-answer tests

These are 180-case exploratory results per arm following failed computation prerequisites. The separate historical Jev 3,600-case study is not this table, and a new 3,600-case main study has not run for the current arms. No threshold was retrospectively selected here to turn a poor fixed-threshold result into a success. Determined-answer tests · Methods

The first menu test used eight underlying examples in two domains, each with a fixed correct answer. Each appeared with the answer present and an escape available, with the answer omitted and an escape available, and with the answer omitted and no escape. The choices were shown in their original order, in reverse order, and under opaque labels whose meanings were explicitly supplied. This produced 24 assigned presentations per condition for each service. All 288 responses were valid. Menu tests

Evidence available: Selection and menu-coverage counts were recomputed from the retained data. The forced-confidence >=.9 result is absent from the full export and comes only from supporting source records; it was not verified by replaying this export. Public source and protocol citations are still pending.

The available-answer and escape decisions were perfect across these presentations. The forced selections did not reach the predeclared 0.9 threshold in either reported confidence or chosen-option probability. This qualifies the historical concern about confidently selecting a false menu option: the interface still cannot express an answer it does not offer, but these fixtures did not produce high-confidence forced errors. Ordinary semantic accuracy is undefined in the forced condition because every legal content answer is false by construction. Menu tests

Menu coverage means that the correct content category appears among the offered choices; the escape does not count as that category. Responses to this coverage question were less reliable than the selections themselves. A model could choose the correct answer or escape while answering the separate question about menu coverage incorrectly. This describes its outputs, not an internal reasoning mechanism. Menu tests

Can it distinguish known absence from insufficient evidence?

The follow-up offered two exits for different situations. In one, the evidence establishes the answer but the menu leaves it out. In the other, the evidence does not establish an answer at all. Twelve matched groups of underlying examples each appeared with the answer present, with a known answer absent, or with genuinely insufficient evidence, alongside plausible content alternatives. All 144 responses were valid. Two-exit tests

Evidence available: All numerical cells were recomputed from the retained condition groups. Public source and protocol citations are still pending.

Every arm identified insufficient evidence correctly on these fixtures. Microsoft and Flash also handled known absence correctly; Clef sometimes chose an inappropriate content answer or confused the exits, while Perplexity's known-absence errors were exit confusions. None of the errors reached the frozen high-confidence or selected-probability threshold. The earlier perfect single-escape result therefore does not transfer uniformly to this richer menu. These finite lookup tasks should not be blended with difficult computational determinacy into a single uncertainty score. Two-exit tests

Does the answer survive an equivalent presentation?

To compare equivalent presentations, the analysis first translated each choice and its probability keys back to their original meanings. It compared the base presentation with a bit complement, renamed symbols, and both changes together. Complementing the bits changed the rule and target as well as the input, preserving the underlying task. A model that gives the same wrong answer across all versions is consistent, but still wrong. Equivalent-presentation tests · Dependence and uncertainty

Evidence available: Complete-group counts were recomputed from exported pairs mapped to common meanings. No missing response was filled in to complete a group. Public source and protocol citations are still pending.

One Flash group is incomplete because the probabilities in its base extraction response failed the normalization check. Its other variants still count in per-case accuracy, but no comparison with a missing valid base response was invented. This cohort did not repeat identical requests as a control. Differences between presentations therefore cannot be divided into effects of the presentation, provider variation and sampling variation. Equivalent-presentation tests · Dependence and uncertainty

Does asking about the menu separately change the answer?

The menu-coverage follow-up selected four underlying examples by a rule that did not depend on their outcomes. For each service, it tested whether the answer was present or absent, asked the coverage question in three ways, and either asked it alongside the selection question or in a separate call. Negative wording was scored in reverse so that all scores referred to the same proposition. The initial allocation contained 192 requests. Another 128 identical-request repeats were planned in advance for the original wording only. All 320 responses were valid. Menu-coverage tests · Shared-context limits

Evidence available: All numerical cells were recomputed from the initial responses and probability spreads in the scheduled repeats. Public source and protocol citations are still pending.

All bundled decision choices were correct across wordings and repeats. Clef's omitted-answer states illustrate why coverage needs its own endpoint: original wording was correct in four of four bundled cases but only one of four standalone cases; reverse-scored negative wording was correct in one bundled case and none standalone. Its original-wording repeated probabilities were identical in all recorded triplets. Microsoft made two initial coverage errors, both with negative wording on omitted-answer states. Menu-coverage tests

Both the combined and separate calls received the full menu and selection rules in their shared context. This differs from the first menu test, and the follow-up also uses a smaller preselected set of examples. Better scores therefore cannot show that wording alone fixed the earlier errors. The repeat rounds ran one after another. Identical recorded probabilities do not prove that the service is always deterministic, and changes between rounds cannot separate time-related drift from sampling variability. Shared-context limits · Menu-coverage tests

The news test uses 550 historical records, with the earliest 330 used to choose calibration settings and the later 220 reserved for testing. The test set has 79 stories labeled important and 141 labeled not important. These are proxy labels: earlier pass/qualify decisions count as important and hold/kill decisions as not important. They came from an editorial/LLM chain, not independent ground truth. The model inputs omit URLs, heuristic scores and labels assigned after adjudication. Calibration choices were fixed before test scoring, and every current-service test response was valid. News classification · Methods

Evidence available: Valid-response counts, accuracy, AUC and Brier error were recomputed from the retained data. True positives (TP), false positives (FP) and false negatives (FN) are absent from the full export and come from supporting comparison records. Public source and protocol citations are still pending.

At the default threshold, the services label too many stories important. A score may still put useful stories nearer the top even when that threshold produces poor accuracy. Neither default accuracy nor always choosing the majority class tells us how well the model allocates limited review time. For automatic routing, we must also count disagreements with the reference labels rather than look only at how few important stories are missed. News classification · News routing

Evidence available: All numerical cells were recomputed from the retained data. These are research-policy scores for the current services, not permission to deploy. Public source and protocol citations are still pending.

For a valid importance probability p and frozen threshold t, route positive when p >= t, negative when p < t and p <= 1-t, and otherwise to fallback. At t = 0.5, the positive route wins the shared boundary. The procedure searched the calibration-only grid 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95, 1.00 for the largest share routed without fallback subject to an important-miss rate no greater than 2.5%, breaking coverage ties toward the lower threshold. The historical selector computes coverage and important-miss rate on valid calibration rows; important misses are proxy-positive rows routed negative, divided by valid proxy-positive rows. Missing or invalid responses always fall back without probability imputation; if no threshold qualifies, no autonomous route is allowed. The saved thresholds are 0.8/0.7/0.75/0.9 in table order and were frozen before test dispatch. All test rows here are valid, so the displayed test important-miss denominator is all 79 proxy-positive stories. News routing · Methods

These scores describe a tested routing policy, not an operating editorial system. Although few proxy-important stories were routed negative, many automatic decisions disagreed with the reference labels. Low important-story miss rates alone do not establish dependable autonomous selection. News routing

Equal review budget

The offline utility analysis selected a 50% review budget using calibration and then compared 110 review slots on the same test split. Equal-budget review

Important stories found at equal review effort

Two separate cohorts, not one ranking: the historical panel uses 110 review slots among 220 items (79 proxy-important); the current-code panel uses 85 among 169 eligible items (59 proxy-important). Stored significance is not current JavaScript. Historical editorial/LLM labels are proxies, not human gold; inputs can differ.

Historical cohort · 110 review slots

Current-code comparison · 85 review slots

Evidence available: The review-budget results are absent from both integrated exports. This table comes from supporting offline-review records and was not verified by a full-suite replay. No time saving was measured. Public source and protocol citations are still pending.

The historical significance path reads Number(significance ?? heuristic_significance ?? 1) / 10; the stored Phase-A baseline reads heuristic_significance / 10. The former was once mislabeled “heuristic alone.” Its stored information may include later changes or information unavailable in the semantic packet. It is an operational historical comparator, not an equal-information causal control. Review-baseline limits

Perplexity retrieved more proxy-important items than historical Jev at this budget but did not demonstrate an advantage over these historical baselines. Its one-story gap from Phase-A is descriptive, not evidence of statistical superiority for the heuristic. That comparison did not execute the current scoring code or measure reviewer minutes. The current-code comparison below addresses the narrower ranking question. Cheap or fast component inference does not measure retrieval, disagreement resolution, fallback, correction or review time. No prospective human-news workflow trial has completed. Equal-budget review · Review-baseline limits · Methods

Does Jev find more important stories than the current rules?

The earlier 220-item comparison used stored historical scores. On October 17, we instead ran the pinned current JavaScript scoring code on recovered original inputs and compared it with native Jev 1.14 on the same source facts. At the prespecified budget of 85 review slots, the rules found 50 of 59 stories labeled important; Jev found 39. All 169 cases were valid in both arms.

Precision—the share of the 85 selected items labeled important—was 58.82% for JavaScript and 45.88% for Jev. Recall—the share of all 59 proxy-important items found—was 84.75% and 66.10%. ROC AUC, which measures ordering across the cohort rather than only the selected queue, was 0.762558 and 0.670493, respectively.

The queues shared 44 items. Among the 41 items selected only by JavaScript, 19 were proxy-important; among the 41 selected only by Jev, 8 were proxy-important and 33 were not. Those eight stories do not establish a benefit from combining the rankers: a hybrid policy would have to account for its additional or reallocated review slots.

For this use case, the result supports retaining the existing rules. Jev returned valid answers throughout, but did not improve this ranking. The comparison does not measure production editorial quality, saved review time or a general limit on decision models. It supplies no demonstrated benefit that justifies replacing the rules or expanding inference now.

How was the current-code comparison made?

We retained 169 unambiguous single-receipt cases from the earlier 220-item cohort and excluded 51 cases with multiple source receipts. Selection order was label-blind, but the exclusions may bias source, topic and importance coverage. Earlier pass/qualify versus hold/kill decisions supply the 59 positive and 110 negative development proxies; historical workflow dependence may favor agreement with those labels.

The comparator executes full current scoring, including upstream magnitude scoring, source adjustments and the final combiner, on original source inputs. It is not a replay of full ingestion, deduplication or relevance filtering. Jev received the original public facts and allowed metadata available to the rules, including metadata the rules did not necessarily use, but no JavaScript output scores or reference labels. Its raw probability of important determines its ranking; the JavaScript ordinal score is not a probability.

The frozen policy sends invalid or missing outputs to review first, then orders valid scores descending, with original date and identifier ascending as tie-breakers. No fallback slots were needed. JavaScript selected 20 of 34 cases tied at its cutoff score of 2; Jev selected both cases at its cutoff of 0.51. Neither the review budget nor tie policy was retuned. The descriptive comparison has no uncertainty interval or causal workflow claim.

Recorded Jev HTTP transactions had a median duration of 0.213588 seconds and a p90 of 0.425340 seconds. These are client transaction times, not isolated inference times or a controlled speed comparison with JavaScript. Actual billing remains unknown; the $16.90 risk reservation is not a bill. Neither timing nor cost changes the ranking result.

The dated aggregate download contains the displayed counts, metrics and source hashes without case rows, source text or private identifiers. Hashes bind the retained audit artifacts; they do not make the underlying private evidence publicly replayable. The figure generator reproduces the plot from that download. No new inference or production change is part of this page update.

Recorded news-call duration

The saved untuned Laya tests failed their competence controls. They also include longer tasks, but the four current services have not completed corresponding longer runs. Better scores on current controls cannot fill that gap. These historical results do not compare an untuned Laya model with a trained checkpoint. Historical Laya tests

Evidence available: These values were recomputed from historical summaries, without case-by-case verification. Laya is not a new fifth service. Public source and protocol citations are still pending.

Plausible applications, not validated deployments from this study. The strongest practical case is a stream of similar decisions whose meanings can be stated clearly, whose errors can be detected, and whose consequences are reversible. A bounded semantic model could suggest the destination of a support ticket, assign a taxonomy label, order a review queue, suggest an appropriate tool, flag possible duplicates, or recommend escalation to a person. Those applications are candidates for evaluation, not a list of tasks this comparison proved safe or economical.

The attraction is the interface. Software can receive a named category or score without extracting it from a paragraph, then apply explicit fallback rules. That can make integration simpler, but the useful unit is the whole workflow. A duplicate flag should invite comparison before records are merged; a tool recommendation should remain subject to permission and input checks; an escalation score should not remove the route to a human. A confident legal category can still be wrong, and a probability over the offered choices cannot certify that those choices cover the situation.

The poor fits follow from those limits. Exact rules that can be executed reliably in code do not need a learned approximation. Missing evidence calls for retrieval or abstention, not a more decisive score. Bad or overlapping menus need redesign and appropriate escape options. Irreversible or costly decisions need protections beyond a bounded answer, while low-volume tasks may never recover the cost of labeling, evaluation and maintenance. Even a useful model can be the wrong engineering choice when a heuristic or ordinary classifier does the job more cheaply and transparently.

The reading and easy-menu results show that these services can perform some bounded tasks. The harder rule failures show why those successes cannot establish reliability on a different task, and the sufficiency results show why recognizing a determined answer must be tested separately from computing it correctly. These findings leave room to evaluate practical uses, without establishing a business case or dismissing semantic classification as a whole.

Training for an organization's decisions need not mean building a foundation model from scratch. Start by comparing the least complicated systems that could do the job. The following are engineering options to evaluate; this study did not measure a training benefit.

- A conventional classifier or embeddings-based baseline. If the categories are stable and labeled examples are available, a small classifier or a classifier over text embeddings provides a useful reference. It tests whether a more elaborate decision model is necessary at all.

- Fine-tuning an existing open-weight model. Adaptation can be investigated when general-purpose behavior misses stable, organization-specific meanings. The relevant experiment is whether a tuned candidate improves the protected evaluation relative to its untuned version and credible alternatives, after accounting for operating costs. Improvement is a hypothesis to test, not a consequence of having run training.

- Distillation from independently audited labels. A stronger model can propose labels or candidate explanations, but those outputs should not become unquestioned truth. Independently review label quality, preserve disagreements and hard negatives, and train a smaller system only on a dataset whose provenance and errors are understood. A model trained to reproduce an unaudited teacher can reproduce its mistakes more efficiently.

Cloudflare's primary announcement says that Clef weights are released under Apache 2.0 and describes fine-tuning services beginning with a hands-on forward-deployed engineering team, followed later by a self-serve platform.[1] This establishes an announced adaptation route; it does not establish general availability of the self-serve platform, customer eligibility, or measured fine-tuning benefit in this study. Jev fine-tunability is not established by the evidence assembled here, so this guidance does not assume it. Historical untuned Laya is not a trained-versus-untuned comparison and cannot estimate what a trained checkpoint would gain.

Training is worth considering when the decision meaning is stable enough to learn, sufficiently specific to the organization to justify adaptation, and frequent enough for a benefit to exceed labeling and maintenance costs. The dataset needs quality labels, negative examples and edge cases, including ambiguous cases and cases that should fall back. Reserve a protected chronological, event-grouped holdout before tuning: related accounts of one event must not leak across training and evaluation. Keep that holdout out of prompt selection, threshold selection and repeated training decisions; later iterations need fresh evaluation windows.

Compare the candidate with a simple classifier, a heuristic and a prompted general model, as well as the untuned model and the actual workflow it would replace. Measure correction effort, monitoring, drift and retraining costs rather than treating inference cost as the whole saving. A failed cellular-automaton prerequisite does not by itself create a need to train. If the intended job is an exact rule, code remains the appropriate baseline; if the intended job is semantic triage, evaluate that job directly. No fine-tuning experiment or measured training gain is reported here.

For Future Shock news ranking, the current evidence supports retaining the existing JavaScript rules. The separate current-code comparison found 50 proxy-important stories in 85 review slots, against Jev 1.14’s 39, on 169 historical items. This is a narrower finding than an end-to-end workflow trial, but it gives no reason to replace the rules. Current-code result and limits

The earlier 220-item comparison remains a separate historical result: at 110 review slots, the stored significance path caught 64 proxy-important stories, the stored Phase-A heuristic 57, Perplexity 56 and historical Jev 49. Those stored-score comparisons had potentially unequal information. The new comparison executes current scoring on recovered source facts, but still uses development proxies rather than fresh independent editorial judgments. Neither comparison measures reviewer time or net workflow benefit.

An advisory ranker would suggest what a reviewer reads first while leaving all items available. It would not decide whether evidence is sufficient, whether a claim is factual, or whether an article should be published. Preserve the separation between creator and validator: a model that proposes a story, label or explanation should not be its own sole source of acceptance. Independent source checking and editorial judgment remain necessary even when a score is high. No single score warrants factual or publication authority.

Proposed acceptance criteria for a separate trial

These are proposed targets for a separate trial, not predeclared gates from the experiments, achieved results, universal standards or authorization to deploy. Their differences reflect the consequences of an error. A poor ordering can be revised, a wrong tag creates cleanup, and an automatically rejected important story may never reach a reviewer. Before a trial, specify those costs and review the targets against them. The historical 95% CA computation gate remains unchanged and separate.

Advisory ranking. Use fresh, independently labeled material with a chronological, event-grouped evaluation design. Compare against the actual current workflow at equal review effort. Require either better important-story recall at that effort, or matched recall with at least a 20% measured reduction in review time, including correction and maintenance effort. The 20% proposal asks for enough margin to justify integration and continuing upkeep; it is not an observed saving. Repeat the comparison across time windows, report uncertainty, and inspect topic and small-source subgroups so that an aggregate gain does not conceal poorer coverage of less prominent sources. Keep items accessible throughout; ranking need not introduce automatic discard.

Low-risk tagging. For any category proposed for automatic assignment, aim for at least 95% precision on the automatically handled subset, with meaningful coverage measured per category and fallback for everything outside that subset. Precision here means the share of automatic labels that independent review judges correct; coverage means the share of eligible items the system actually handles automatically. Specify useful coverage and uncertainty requirements before evaluation, rather than achieving precision by labeling almost nothing or hiding a weak category in an aggregate. The proposed precision target limits routine cleanup for reversible labels; it is not the historical CA accuracy gate and has not been achieved for an FS tagging deployment in this study.

Automatic rejection, if ever separately justified. Require an important-story miss rate below 1%, supported by an independently labeled sample and uncertainty analysis. Define the denominator as all independently judged important stories in the evaluated stream, and the numerator as those routed to automatic rejection. The stricter proposal reflects the cost of losing an item before anyone reads it. Zero observed misses in a small sample is not proof of safety; sample size, event dependence and uncertainty must support the claim. There is no need to add automatic discard merely because a ranker is useful, and this page does not recommend deploying it now.

Factual validation and publication. None of those targets would authorize a model to validate its own evidence or publish an article. Those responsibilities require independent validation and an explicitly authorized editorial process, not a higher confidence threshold. A favorable shadow result would justify reviewing the next bounded experiment, not quietly widening the model's authority.

A future proposal would need a specific, independently motivated benefit and a fresh prospective evaluation of the combined workflow, including evidence preparation and correction effort. The criteria above describe what such a proposal would have to establish; they are not a recommendation to launch another trial on the strength of this result.

The linked Cloudflare source is a vendor primary announcement, used only for the Apache 2.0 weights and announced fine-tuning route. Its benchmark, speed and training-benefit claims are not adopted as findings of this study. The applications, training choices and deployment targets above are recommendations; the historical measurements retain their original evidence boundaries below and in the preceding results.

[1] https://blog.cloudflare.com/clef-decision-models — Introducing Clef: our open-source decision models, and new RL fine-tuning platform

Evidence available: The full export does not include billing, token or reserve quantities. This retained historical snapshot is neither a current bill nor a new replay measurement. Its 6798 reserves cover 6796 task instances plus two smoke attempts. Public source and protocol citations are still pending.

Known reported billing is only the known portion of the bill. Unknown usage/billing remains unknown, and conservative cap accounting is a separate quantity. The cumulative total includes the two smoke attempts and does not count inherited pilot responses twice. It does not aggregate all historical Jev or Laya work into an all-time total. The historical reserve snapshot reconciles to 6,796 verified task instances plus two smoke attempts. Private accounting evidence is not distributed. Cost records

Response types, scores and thresholds

CHOICE supplies a probability distribution over the offered menu; NOUL supplies a probability-like score for an absolute proposition, a statement evaluated in its own right rather than as one choice among alternatives. Reported confidence, selected-option probability, absolute proposition probability and token likelihood are distinct measurements. Token likelihood concerns the probability of output text; it is not interchangeable with a probability that a decision is correct. Binary bit and sufficiency predictions use p >= 0.5. Coverage reverses negative-wording probability as 1 - raw before applying that threshold. The coverage false-certainty rule is signed p >= 0.9 for absent answers or p <= 0.1 for present answers. Menu and exit false-high-confidence checks use the predeclared 0.9 threshold separately for reported confidence and selected-option probability. Methods · Menu tests · Two-exit tests · Menu-coverage tests

AUC reports ranking, with tied positive/negative pairs receiving half credit in the historical recomputation. Brier error is squared probability error against the binary reference. Accuracy and joint correctness retain assigned denominators for demonstrated successes; probability metrics exclude invalid observations and state the smaller valid denominator. AUC with only one truth class and forced-menu semantic accuracy are undefined, not zero. News references are editorial proxies; deterministic task oracles (reference answers computed from the task rules) and editorial labels are not interchangeable. Determined-answer tests · News classification · Methods

Amendments, errors and no retry

A documented prospective successor accepted any exact co-maximum option rather than requiring the first declared maximum. The original stop and settlement were retained, and the accepted policy was applied through a hash-bound overlay; near ties were not made exact. Clef Flash's distribution summing to 1.0003 remains invalid against the frozen 0.0002 normalization tolerance. It was not renormalized. Two Perplexity rate-limit failures remain failures. Methods

Failed requests were not retried. Approved continuations inherited authenticated prior outcomes and called only untouched assignments. Planned coverage repeats are a different scientific object: each has a distinct trial identity despite identical wire bytes and was fixed before outcomes were observed. No best-of-N selection or cache collapse is appropriate. The reference answer belongs in scoring, not in the input given to the model. Methods · Menu-coverage tests

Dependence and uncertainty

The paired-analysis intervals use 95% percentile cluster bootstrapping with 3,000 draws, fixed seed 20261014 and stable per-table streams. The bootstrap repeatedly samples whole groups of related examples, retaining their variants and invalid responses together rather than treating each as independent. Pooled representation rows use 40 seed groups and family rows eight; the coverage diagnostic has only four source seeds. These are descriptive intervals for a small development diagnostic, with many exploratory contrasts, not confirmatory tests. Boundary intervals do not prove certainty. Full saved interval fields are available in the internal paired-analysis source rather than selectively promoted in the main table. Dependence and uncertainty

News stories can describe the same event, so chronology alone does not establish event independence. This page supplies no new significance test or unmeasured paired uncertainty for the news-budget comparison. Follow-up designs were informed by earlier observations, and packaging the record does not make those design choices retrospectively preregistered. News classification · Equal-budget review · Reproduction and release limits

The retained evidence includes the historical offline suite, its reconstruction, sanitized aggregates and the earlier Jev prerequisite integration. Those earlier checks do not independently validate this edited page. The new Jev prerequisite attempts are reported separately and do not change the historical four-service inventory.

Historical Jev news calibration remains qualifies=false, meaning it did not qualify for autonomous routing. Laya is available only as summary results. The other Jev 1.14 task families in the original suite have not run; their results are unavailable, rather than filled with zeros or borrowed from historical Jev. The separate October 17 current-code comparison reports its own news cohort and protocol above.

The six October 16 downloads retain their original bytes and bounded publication clearance. The dated October 17 reader copy and aggregate download cover the current-code ranking update separately. Descriptive evidence links point to related sections of this briefing; public historical source and protocol citation gaps remain. Aggregate downloads and source hashes do not provide the private case records needed to replay the raw calls. No external academic peer review is claimed.

The original methodology and data dictionary remain available with the frozen downloads. For the current-code comparison, the dated aggregate JSON and Python figure generator reproduce the displayed plot without provider calls; the underlying request and response audit still depends on retained private records.

Source/protocol public citation pending. Historical provider/time/renderer confounds remain; no controlled causal superiority claim.

The new native TypeSafe Jev 1.14 run reproduced all 60/60 supplied cell values correctly, passing the reading test. It answered 356/529 one-step cases correctly, with 527 valid responses and 2 invalid responses, failing that prerequisite. The predeclared gate requires every response to be valid, at least 95% accuracy overall and at least 95% in each class. Passing reading alone does not qualify the model for the downstream tests.

Comparing the same cases across historical runs

These are previously used development cases, not fresh held-out trials. The versions were tested with different renderers, providers and collection times, so a score difference cannot be attributed to the version change or establish a general regression. The logical cases match, but that does not make the HTTP requests or tokenization identical. Class-specific outcomes shift toward more positive predictions. No significance or population-generalization claim is made.

Failed responses and recorded usage

One HTTP 529 failure has unknown usage, not zero tokens. One HTTP 200 distribution summed to 0.99 and remains invalid under the frozen 0.0002 tolerance; its known usage is included. Both invalids remain assigned and incorrect; neither supplies an imputed probability. No retry, salvage, replacement or tolerance adjustment is implied.

Observed subtotals: 639,336 input and 50,032 output tokens across 588 usage-known responses; 1 response has unknown usage. These exclude the earlier separate compatibility probe and are not complete token totals or a bill.

Timing for each phase is available in the aggregate JSON. The timers start with HTTP transport and end when the raw event is constructed, before it is written durably. They exclude startup waits and the pacing between calls, which differed across phases. These timings therefore do not measure provider computation alone or end-to-end throughput, support a controlled speed comparison, or guarantee future latency.

Remaining local allowance $0.029600058 is insufficient for the next $0.10 attempt. This is conservative risk-allowance accounting, not verified pricing, spending or an invoice guarantee; billing is unknown. No extra inference or cap change is authorized.

The original prerequisite run completed 589 reading and one-step attempts; these are calls, not additional distinct benchmark cases. Within that original suite, Jev 1.14 sufficiency, news, domain/representation, menu/escape, exit and coverage tests remain unrun (the separate current-code news-ranking comparison above uses its own 169-item protocol), with unavailable results represented as null metrics. Including these prerequisite results on the page does not complete the Jev suite or authorize publication.