Methods FAQ

This page is seeded from the engine’s fixture register — the F-numbered fixtures the engine is held to on every build — and from the cross-checks against reference implementations. Each entry below is read from the same generated report the validation page renders, so the status shown here is the status shown there, and an expected value that has not yet been captured from its primary source carries its [unverified] marking here exactly as it does there. Nothing on this page is a result until that page says so.

The whitepaper that derives each method, /methods, is a later build day. Until it exists, the validation page is the reference for which method is used where.

Conventions that are ProofPack’s, not a regulator’s

The engine’s scope statement lists these as ProofPack reporting conventions, and every pack captions them as such where they appear:

• Wilson intervals, no continuity correction, for every proportion, with the Clopper-Pearson interval shown alongside when the count is 0 or n. The Wald interval is computed only inside tests, to assert that it is never emitted (fixture F1).
• Low-n precision tiers: every proportion carries a tier label derived from the Wilson half-width at its sample size (fixture F8). The sample-size figure that could be derived from the same arithmetic is asserted in tests and never rendered in a pack: ProofPack does not give sample-size advice.
• The bootstrap: percentile intervals, B = 2000 and seed 20240101 by default, numpy.random.default_rng(seed), stratified by class; declared values are recorded in the manifest next to whether they were declared or defaulted. On clustered data (clustering.unit: case_id) the cluster bootstrap replaces the analytic interval, and the analytic method is refused rather than silently applied (fixture F9).
• PPA/NPA routing: with reference_standard.type: comparator the pack reports positive and negative percent agreement and the words sensitivity and specificity are absent from the output (fixture F11), following FDA’s 2007 statistical guidance on diagnostic tests (final).
• Indeterminates both ways: with indeterminates.policy: report_both_ways, the as-positive and the as-negative tables are both rendered (fixture F10).
• The heterogeneity test is a footnote, never a headline (fixture F6).

The fixtures

What does fixture F1 check?

Wilson and Clopper-Pearson intervals at 81/263, and Wald never emitted.

Oracle: statsmodels.proportion_confint · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F2 check?

2x2 panel: sensitivity, specificity, PPV, NPV, LR+, LR-, DOR, Youden, F1, MCC, and PPV/NPV at a declared prevalence.

Oracle: R2 section 9 F2 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F3 check?

AUROC, DeLong standard error, logit interval when the Wald interval leaves [0,1], paired DeLong comparison, AUPRC and the stratified bootstrap on a pinned seed.

Oracle: scikit-learn roc_auc_score; R2 section 9 F3 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F4 check?

Calibration: Brier, IPA, observed-to-expected ratio, calibration intercept and joint slope, and ECE at two bin schemes.

Oracle: statsmodels GLM; R2 section 9 F4 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F5 check?

Version comparison on a paired table: exact McNemar, corrected chi-square, paired accuracy difference and the Newcombe paired interval.

Oracle: statsmodels mcnemar · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F6 check?

Subgroup sensitivity by site with the heterogeneity test reported as a footnote only, and Holm adjustment.

Oracle: R2 section 9 F6 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F7 check?

Drift: population stability index and the two-sample Kolmogorov-Smirnov test on pinned seeds.

Oracle: R2 section 9 F7 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F8 check?

Precision tiers: Wilson half-width at three sample sizes; the derived sample-size figure is asserted in tests and never rendered in a pack.

Oracle: R2 section 9 F8 · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F9 check?

Clustered data: DeLong is refused rather than silently applied, and the cluster bootstrap reproduces the independent-case interval.

Oracle: Hand-derived from F3 · Tolerance: 0.01 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F10 check?

Indeterminate results analysed both ways, with both tables rendered.

Oracle: Hand arithmetic; FDA 2007 statistical guidance (final) · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F11 check?

PPA/NPA routing when the comparator is not a reference standard, with the words sensitivity and specificity absent from the output.

Oracle: FDA 2007 statistical guidance (final) · Tolerance: exact string assertion · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F12 check?

Every HALT gate H01-H12 fires on its fixture, exits 3 and writes no document.

Oracle: Master Spec section 10 item 4 · Tolerance: exit code and file-system assertion · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F14 check?

Newcombe method 10 unpaired difference, cross-checked against the worked example transcribed from Newcombe (1998) Table II.

Oracle: fixtures/newcombe_table2.json, transcribed from the paper in session · Tolerance: 4 decimal places · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F15 check?

Sample-size-aware critical value for the population stability index at two sample sizes.

Oracle: scipy.stats.chi2.ppf · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F18 check?

Synthetic story cohort: an injected subgroup gap is detected in at least 90% of 50 seeds, and an injected miscalibration is detected.

Oracle: Generator with a known injected effect · Tolerance: detection rate · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F19 check?

Egress: the telemetry payload validates against egress_schema.json, a small intersectional cell is suppressed, a real site name appears in no egress byte, and --offline completes with the network namespace removed.

Oracle: egress_schema.json; run under unshare -rn · Tolerance: byte-level assertion · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F20 check?

Licence verification: valid, expired, tampered, wrong-key and trial licences each produce the right refusal or watermark.

Oracle: scripts/verify_licence.py, an independent reference verifier · Tolerance: exact · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

What does fixture F21 check?

Regression snapshot on the public per-stay sepsis sample: overall and per-site JSON frozen, any later change explained in the changelog.

Oracle: Frozen snapshot · Tolerance: byte-identical or explained · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

Cross-checks against reference implementations

Wilson and Clopper-Pearson intervals against statsmodels.proportion_confint

Oracle: statsmodels · Tolerance: 1e-4 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

AUROC against scikit-learn roc_auc_score

Oracle: scikit-learn · Tolerance: 1e-6 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release.

AUROC, the DeLong interval and the paired DeLong test against R pROC on the aSAH dataset

Oracle: R pROC::roc, ci.auc(method=delong), roc.test · Tolerance: 1e-6 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The R capture has not been made. The engine's own test for it is marked expected-to-fail with the reason 'pending R capture on day 12 (pROC on aSAH); fixture not yet committed', and fixtures/r/proc_asah.json does not exist in the engine repository.

Calibration intercept and slope against R rms::val.prob on the committed calibration vectors

Oracle: R rms::val.prob · Tolerance: 1e-6 · Status on the validation page: Pending

No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The R capture has not been made.

Clustered AUROC standard error against an R reference implementation

Oracle: R package not yet named · Tolerance: not yet set · Status on the validation page: Pending [unverified]

No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: D2 section 5.4 records the R package for this comparison as still to be chosen. Nothing has been fetched or read to choose it, so the row is marked [unverified] and stays marked until it is.

Questions the fixtures answer

Why Wilson and not Wald? Because the Wald interval collapses to a point at 0/n and n/n and can leave [0, 1]; fixture F1 asserts the engine never emits one. Where a reviewer expects the exact interval, Clopper-Pearson is shown alongside at the boundary counts.

Why is DeLong refused on my clustered data? Several rows per case are not independent, and DeLong’s variance assumes they are: fixture F9 shows the naive standard error shrinking by the square root of the cluster size. The engine takes the cluster bootstrap instead and says so in the pack; it does not offer the analytic interval as an option.

Why is PPV shown twice? Once at the test set’s observed prevalence and once at each intended-use prevalence you declared (fixture F2). If the two differ by more than 0.10 absolute the pack carries flag H10, which is a finding, not a halt.

Why does the pack say “not assessable” instead of a number? Because the engine hit a typed reason it will not talk past: calibration with a score that is not a probability, AUROC with too few positives or negatives, a suppressed cell, an interval it cannot estimate, or a comparison that is not like-for-like. The reason is printed beside the words. There is no silent fallback to another method.

Why are there no numbers on this page? Because none would come from the engine. Expected values live in the engine’s fixture files and are compared in CI; the validation page publishes the comparison per release. A number that appears in these docs without an interval, a method and an n does not come from the engine.