Methodology and public receipts

01 / Research question

How much does sharing the same recoverable instruction or evidence error increase fleet failure dependence relative to distributing severity-matched errors independently, while holding the underlying task and baseline difficulty fixed?

The broader retrospective layer contains 395 models on 27,842 exact common items. Its execution-side selection audit reported 3,253 support-qualified model pairs from 77,815 possible unordered pairs (4.18%). The public Stage 1 bundle does not include the event-level pair-support matrix needed to regenerate that selection count.

02 / Endpoint populations

The legacy population contains nine endpoints from seven developer families. The original ten-endpoint execution failed its frozen infrastructure-support gate; Qwen3-30B/Alibaba was excluded in whole before outcome access. The first nine-endpoint continuation also failed its unchanged gate. The reported execution preserves those terminal cells and reopens only retryable transport errors under an outcome-blind five-attempt, low-concurrency recovery amendment.

The contemporary population contains six endpoints from six developer families, frozen before legacy outcomes.

Legacy and contemporary populations are analyzed separately and are never pooled. Fallback routing is disabled for all endpoints.

The contemporary model and provider names are the frozen route labels recorded by the package. They are provenance claims about the requested routes, not independent validation of the underlying model weights.

03 / Six experimental conditions

All conditions retain the same correct answer and deterministic scorer. Models are told that reviewer notes and evidence are untrusted and may contain one error; the operative task is always to derive and return the correct answer.

04 / Estimand definitions

Dependence inflation (DI)

Plain language: DI measures how much the fleet's actual failure variance exceeds what would be expected if every endpoint failed independently at its own observed rate. A DI of 1.0 means independence; values above 1.0 mean failures are more clustered than independence predicts.

Formal: For condition c, let Σ_c be the sample covariance matrix of endpoint failure vectors with equal weights w_m = 1/M. Fleet variance is V_c = w′Σ_cw. Independent-diagonal variance is D_c = Σ_m w_m² Var(Y_imc). Then DI_c = V_c / D_c.

DI ratio (primary estimand)

Plain language: the DI ratio compares the dependence inflation when all endpoints receive the same error with the dependence inflation when each endpoint receives a different error of matched severity. Values above 1.0 mean sharing the same error increased dependence.

Formal: θ_E = log(DI_{E_shared} / DI_{E_independent}). Report exp(θ) as the ratio.

Absolute volatility ratio (AVR)

Plain language: AVR is the ratio of fleet standard deviations between shared and independent arms. It captures whether shared errors make fleet-level outcomes more variable.

Formal: AVR = √(V_shared / V_independent).

Mean loss difference

Plain language: the difference in average fleet failure rate between shared and independent arms. Reports whether sharing the error changes how often the fleet is wrong on average, separate from whether it changes how clustered the errors are.

05 / Support quality tiers

Both legacy primary contrasts are classified estimable_degraded, not confirmatory_quality. The evidence-corruption contrast has 665/800 (83.1%) matched support overall; two domains (arithmetic 71.9%, extraction 76.9%) fall below the 80% per-domain confirmatory floor. The instruction-anchor contrast has 682/800 (85.3%) overall.

Both contemporary contrasts achieve confirmatory_quality support (94/100 and 97/100), but their intervals include no change.

06 / Missingness and bounds

Infrastructure failures, ambiguous deliveries, route mismatches, and scorer failures are never recoded as model failures. Missing cells are excluded, and sharp binary-fill bounds cover mean loss and fleet tail events. Primary contrasts use exact matched support: an item contributes only if all endpoints completed both conditions in a pair.

Legacy evidence-corruption mean-loss bounds straddle zero: [−0.0140, +0.0115]. Legacy instruction-anchor bounds also straddle zero: [−0.0099, +0.0149]. Terminal error types in the legacy execution include ConnectError (1,825), HTTP 429 (312), ReadError (239), RemoteProtocolError (186), TimeoutError (31), and orphaned-started attempts (63).

07 / Diagnostic failures

Neutral manipulation check

The neutral manipulation check failed in both populations. The prespecified criterion requires O–N mean failure within ±0.03 and DI_N/DI_O within [0.90, 1.10].

• Legacy: O–N mean difference = −0.0398 (outside ±0.03 margin). DI ratio = 0.995 (within margin).
• Contemporary: O–N mean difference = −0.0333 (outside ±0.03 margin). DI ratio = 1.012 (within margin).

The N rewrite is systematically harder than O by approximately 3–4 percentage points. This limits total anchor-versus-neutral interpretation. It does not directly test the shared-versus-independent ratio, but it weakens confidence that the transformation pipeline changed only the intended feature.

Absent contemporary canaries

The contemporary population has zero canary observations across all six endpoints and all three epochs (start, midpoint, end). The prespecified drift-detection mechanism was entirely absent. No canary-based claim about route or behavior drift is available.

Endpoint degeneracies

• Legacy — Claude 4.5 Haiku: 100% failure rate under both A_shared (787/787) and A_independent (789/789). This endpoint is a zero-variance contributor to the primary anchor DI contrast, reducing the effective anchor fleet to eight endpoints (28 nondegenerate pairs).
• Contemporary — Nemotron 3.5: 97–100% failure rate across all conditions (97/99 even in the O condition). This endpoint is degenerate in both evidence conditions, reducing the effective evidence fleet to five endpoints (10 nondegenerate pairs).

Singleton-stratum correction

In the contemporary bootstrap, two strata contained a single item-family cluster each. A postcollection implementation correction treats these as self-representing certainty strata. This correction was made after legacy results existed and is explicitly not described as outcome-blind. It changes bootstrap execution but no contemporary point estimate, item, verdict, stratum, or estimand.

08 / Summarized amendment register

Twenty named protocol amendments were made between contract freeze and final analysis. Most are claimed outcome-blind at their point of application; the singleton-stratum correction explicitly is not. The amendment chain is self-attested through frozen artifacts and SHA-256 hashes, not independently witnessed.

09 / Evidence register

10 / Stage 1 reproducibility

The first public evidence bundle is a Stage 1 release. Stage 1 means report-level inspection and receipt reconstruction are supported. It does not mean full independent reproducibility: the historical model calls cannot be replayed, some early collection-time patch state is unavailable, and restricted Phase-1 source bytes are excluded pending redistribution permission.

The delivered package establishes internal consistency through SHA-256 checksums, completion receipts, and a manifest. It does not establish bit-exact external reproducibility. The study contract, estimands, cluster definitions, bootstrap parameters, and sample size were frozen before model calls, with simulation-validated operating characteristics (type I error, coverage, power) preceding any data collection.

11 / Inference

Primary common-shock intervals use a 9,999-resample studentized maximum-absolute-statistic bootstrap, clustering by item family within frozen domain and predicted-difficulty strata. The joint coverage target is 95% simultaneous across both primary contrasts (θ_A and θ_E).

The bootstrap was simulation-validated before any pilot call against targets of ≥ 90% power for a DI ratio of 1.25 and ≥ 80% power for 1.20, with familywise two-sided α 0.05. Simulation selected 800 confirmatory items. It did not change using observed effect sizes.

If DI or its ratio is undefined (e.g. zero-variance endpoint), the condition is reported as floor/ceiling saturated with mean and tail loss only. No post-outcome regularization is applied.

← Return to main briefing