Validation of the validator
ProofPack computes the numbers a regulator reads, so it has to show its own working. This page lists every check the engine is held to, whether that check has been run, and what independent implementation it is compared against. It is regenerated from the engine's test artefacts, not written by hand.
Page last updated 2026-09-10.
Nothing here has been run yet. No CI artefact has been ingested. This page is rendering the committed sample artefact, so every row below is shown as pending and no result is claimed for any check. It is published in this state deliberately: an empty validation page is honest, and a full one that has not been run is not.
Where this stands
- 0 passing
- 0 failing
- 40 not yet run
- 40 checks in total
- 4 carrying an
[unverified]marking
| Where these results came from | |
|---|---|
| Artefact kind | sample — not a test run, so no result is claimed |
| Engine repository | JoshSandhu/proofpack |
| Engine commit | none recorded |
| CI run | none recorded |
| Engine version | none recorded |
| Artefact generated | 2026-09-10T00:00:00Z |
| Artefact file | src/data/fixtures_report.sample.json |
This is the committed SAMPLE artefact, not a test run. It exists so the page, the generator and the checks can be built and reviewed before the engine's CI emits a real fixtures_report.json. Every row in it is not_run or not_captured, and the renderer forces every row of a non-CI artefact to PENDING in any case. Row identities come from D1 section 3.2 (the fixture register) and D2 section 5.4 (the validation-page sections); no expected or observed VALUE is written here, because the site never types an engine number by hand.
A check appears here as passing only when a real continuous-integration run says so and records the commit it ran against. A check that is skipped, expected to fail, not yet captured or not yet written is listed as pending, with the reason, in the same table as everything else. It is never quietly omitted, and its expected value is never shown as though it were a result.
Four things about that run are checked in code rather than taken on trust: the commit must be a full forty-character object id and must resolve in the engine repository named above; the run link must be a run of that same repository; the artefact committed in this repository is pinned to “sample”, so a real result has to be fetched from a run rather than typed here; and a real artefact more than 90 days old fails the build rather than sitting here showing last quarter’s passes.
Fixture results
These are the numeric fixtures the engine is required to reproduce. Until the engine's CI publishes a report, no result is shown for any of them.
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
F1 | Wilson and Clopper-Pearson intervals at 81/263, and Wald never emitted No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F1, F1b, F1c, F1d | statsmodels.proportion_confint | 1e-4 | — | Pending |
F2 | 2x2 panel: sensitivity, specificity, PPV, NPV, LR+, LR-, DOR, Youden, F1, MCC, and PPV/NPV at a declared prevalence No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F2 | R2 section 9 F2 | 1e-4 | — | Pending |
F3 | AUROC, DeLong standard error, logit interval when the Wald interval leaves [0,1], paired DeLong comparison, AUPRC and the stratified bootstrap on a pinned seed No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F3 | scikit-learn roc_auc_score; R2 section 9 F3 | 1e-4 | — | Pending |
F4 | Calibration: Brier, IPA, observed-to-expected ratio, calibration intercept and joint slope, and ECE at two bin schemes No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F4 | statsmodels GLM; R2 section 9 F4 | 1e-4 | — | Pending |
F5 | Version comparison on a paired table: exact McNemar, corrected chi-square, paired accuracy difference and the Newcombe paired interval No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F5 | statsmodels mcnemar | 1e-4 | — | Pending |
F6 | Subgroup sensitivity by site with the heterogeneity test reported as a footnote only, and Holm adjustment No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F6 | R2 section 9 F6 | 1e-4 | — | Pending |
F7 | Drift: population stability index and the two-sample Kolmogorov-Smirnov test on pinned seeds No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F7 (v1.1 feature) | R2 section 9 F7 | 1e-4 | — | Pending |
F8 | Precision tiers: Wilson half-width at three sample sizes; the derived sample-size figure is asserted in tests and never rendered in a pack No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F8 | R2 section 9 F8 | 1e-4 | — | Pending |
F9 | Clustered data: DeLong is refused rather than silently applied, and the cluster bootstrap reproduces the independent-case interval No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F9 | Hand-derived from F3 | 0.01 | — | Pending |
F10 | Indeterminate results analysed both ways, with both tables rendered No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F10 | Hand arithmetic; FDA 2007 statistical guidance (final) | 1e-4 | — | Pending |
F11 | PPA/NPA routing when the comparator is not a reference standard, with the words sensitivity and specificity absent from the output No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F11 | FDA 2007 statistical guidance (final) | exact string assertion | — | Pending |
F12 | Every HALT gate H01-H12 fires on its fixture, exits 3 and writes no document No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F12 | Master Spec section 10 item 4 | exit code and file-system assertion | — | Pending |
F14 | Newcombe method 10 unpaired difference, cross-checked against the worked example transcribed from Newcombe (1998) Table II No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F14 | fixtures/newcombe_table2.json, transcribed from the paper in session | 4 decimal places | — | Pending |
F15 | Sample-size-aware critical value for the population stability index at two sample sizes No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F15 (v1.1 feature) | scipy.stats.chi2.ppf | 1e-4 | — | Pending |
F18 | Synthetic story cohort: an injected subgroup gap is detected in at least 90% of 50 seeds, and an injected miscalibration is detected No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F18 | Generator with a known injected effect | detection rate | — | Pending |
F19 | Egress: the telemetry payload validates against egress_schema.json, a small intersectional cell is suppressed, a real site name appears in no egress byte, and --offline completes with the network namespace removed No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F19; Master Spec section 10 item 8 | egress_schema.json; run under unshare -rn | byte-level assertion | — | Pending |
F20 | Licence verification: valid, expired, tampered, wrong-key and trial licences each produce the right refusal or watermark No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F20. The site half of this is built and green; the engine half has not run. | scripts/verify_licence.py, an independent reference verifier | exact | — | Pending |
F21 | Regression snapshot on the public per-stay sepsis sample: overall and per-site JSON frozen, any later change explained in the changelog No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F21 | Frozen snapshot | byte-identical or explained | — | Pending |
Source: Engine pytest run to fixtures_report.json. Row identities from D1 section 3.2.
Cross-checks against reference implementations
Independent review here is done by code, not by invoices: every statistic is recomputed by a library or a language we did not write and compared to ours. The engine’s own test suite already runs comparisons of this kind on every commit; nothing on this page reports a result from them, because no CI artefact has been ingested. "Not yet run" here means "not yet run for a published release", not "never run".
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
xc-proportions-statsmodels | Wilson and Clopper-Pearson intervals against statsmodels.proportion_confint No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | statsmodels | 1e-4 | — | Pending |
xc-auroc-sklearn | AUROC against scikit-learn roc_auc_score No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | scikit-learn | 1e-6 | — | Pending |
xc-delong-proc | AUROC, the DeLong interval and the paired DeLong test against R pROC on the aSAH dataset No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The R capture has not been made. The engine's own test for it is marked expected-to-fail with the reason 'pending R capture on day 12 (pROC on aSAH); fixture not yet committed', and fixtures/r/proc_asah.json does not exist in the engine repository. Evidence: Engine test node id tests/test_discrimination.py::test_f13_matches_proc_on_the_asah_dataset; the engine records XFAIL for it at commit 3bee15c3 on 2026-09-10; D1 section 3.2 F13 | R pROC::roc, ci.auc(method=delong), roc.test | 1e-6 | — | Pending |
xc-calibration-rms | Calibration intercept and slope against R rms::val.prob on the committed calibration vectors No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The R capture has not been made. Evidence: D1 section 3.2 F13b | R rms::val.prob | 1e-6 | — | Pending |
xc-clustered-auroc | Clustered AUROC standard error against an R reference implementation [unverified] No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: D2 section 5.4 records the R package for this comparison as still to be chosen. Nothing has been fetched or read to choose it, so the row is marked [unverified] and stays marked until it is. Evidence: D2 section 5.4 | R package not yet named | not yet set | — | Pending |
Source: Nightly GitHub Actions job in a rocker/r-ver image for the R oracles; the Python oracles run in the ordinary test job.
Published 510(k) and De Novo subgroup tables
These sponsors did not use ProofPack; their public tables are used only as reference shapes and arithmetic checks.
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
pub-k240860 | Sensitivity with a printed confidence interval, recomputed from the printed counts (candidate submission K240860) [unverified] No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The submission number and the figures beside it are transcribed from research note R3 section b and were NOT re-fetched from the FDA 510(k) database in this session. The [unverified] marking stays on this row, and no number from the summary is shown here, until someone opens the primary document and records what it says. Evidence: R3 section b via D2 section 5.4 | — | — | — | Pending |
pub-den230008 | Subgroup rows by skin type, recomputed from the printed counts (candidate submission DEN230008) [unverified] No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: Transcribed from research note R3 section b; not re-fetched from the primary decision summary in this session. Evidence: R3 section b via D2 section 5.4 | — | — | — | Pending |
pub-k252360 | Subgroup table layout used as a golden-document rendering target (candidate submission K252360) [unverified] No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: Transcribed from research note R3 section b; not re-fetched from the primary decision summary in this session. Evidence: R3 section b via D2 section 5.4 | — | — | — | Pending |
Source: Public FDA decision summaries. Where a summary prints counts with an interval, the interval is recomputed from the printed counts and the agreement, and the interval method it implies, are shown.
Adversarial corpora
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
adv-claim-binding | Claim-binding checker: at least 50 narratives that are numerically correct but say something false are all rejected No CI artefact has been ingested, so this check has not been run for a published release. Evidence: Master Spec section 10 item 5 | — | — | — | Pending |
adv-mapping | Mapping corpus: column-name variants map to the right canonical field or halt No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | — | — | — | Pending |
adv-halt-fuzz | HALT-gate fuzzing never yields a silently wrong number No CI artefact has been ingested, so this check has not been run for a published release. Evidence: Master Spec section 10 item 4 | — | — | — | Pending |
Source: Engine test suite.
Determinism and platform parity
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
det-manifest-hash | Two runs on the reference image produce an identical manifest hash and byte-identical JSON No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F17 | — | — | — | Pending |
det-pyodide-parity | Pyodide and native agree on the numeric fixtures within the stated tolerances No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D1 section 3.2 F16 | — | 1e-9 closed form, 1e-6 iterative, reported rounding for bootstrap | — | Pending |
det-arch-parity | arm64 and amd64 agree within tolerance, with the maximum absolute deviation per module tabulated No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | — | — | — | Pending |
Source: Engine release job.
Independent review by people
Nobody has reviewed ProofPack's regulatory wording yet. This section exists so that the empty slot is visible rather than absent.
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
rev-design-partner | Design partner's regulatory-affairs lead reviews the guidance map, the pack cover and disclaimer, the scope section and the outreach templates No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: No design partner has been recruited yet. Evidence: Master Spec section 10 item 18 | — | — | — | Pending |
rev-iq-oq | A regulatory-affairs lead completes the installation and operational qualification script, including the mapping-verification step, inside 45 minutes No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The qualification script is a later build day and no partner has run it. Evidence: Master Spec section 10 item 19 | — | — | — | Pending |
rev-ra-consultancy | Partner regulatory-affairs consultancy comments on the guidance wording No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: No consultancy has agreed to review. Evidence: Master Spec section 10 item 18 | — | — | — | Pending |
Source: Design partner and partner regulatory-affairs consultancy. This is review, not certification.
Per-release artefacts
| Check | What it asserts | Compared against | Tolerance | Result | Status |
|---|---|---|---|---|---|
rel-validation-report | Downloadable validation report for the release No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | — | — | — | Pending |
rel-sbom | CycloneDX software bill of materials for the runner image No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.5 | — | — | — | Pending |
rel-trivy | Container and dependency vulnerability scan summary, with no unpatched critical findings No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: No scanner runs in either repository's CI yet. Evidence: Master Spec section 10 item 17 | — | — | — | Pending |
rel-zap | Baseline dynamic scan of the site and its functions, with no high findings No CI artefact has been ingested, so this check has not been run for a published release. Artefact's own note: The site is not deployed, so there is no URL to scan. Evidence: Master Spec section 10 item 17 | — | — | — | Pending |
rel-iq-oq-script | Installation and operational qualification script shipped with the release No CI artefact has been ingested, so this check has not been run for a published release. Evidence: D2 section 5.4 | — | — | — | Pending |
Source: Release job outputs, downloadable beside each release.
How this page is made, and how you could catch us out
The engine's continuous integration publishes afixtures_report.json artefact for each release. A script in the site repository turns that artefact into the table above; the build regenerates it and fails if the committed output differs by a byte, so a result cannot be typed in by hand. The artefact contract is published asfixtures_report.schema.json alongside it.
Three rules are enforced in code rather than by good intentions. First, a row can show a pass only if the artefact came from a real continuous-integration run with a commit recorded; an artefact that is merely a sample has every row forced to pending, whatever it claims. Second, on any row that is not a pass or a fail theResult column is blank — the generator drops the measured value rather than the page hiding it. Third, that row may not smuggle the result into the text beside it: on a row that did not run, an outcome word from a fixed list — matched,agreed, reproduced, observed and the like — or the literal CI or confidence interval, standing in the same clause as a number, fails the build wherever it appears in that row or in the note above its table. It is a list of words and it is not complete: 95% interval is not on it.
What those two columns do carry is a specification, not a result: what the check will be compared against, and how close it has to get. That is why a pending row still shows a tolerance of 1e-4.
Every cell of every row above is compared, character for character, against the generated file this page is built from: theCheck column, the What it asserts column together with the reason and the evidence printed under it,Compared against, Tolerance, Result andStatus. So is the note above each table and the note under the summary. The build fails if any of them differs by a character, if a row or a note appears that the generated file does not contain, or if a row, a cell or a note is rendered without the marker that comparison uses — so an extra row, or an extra bold line above a table, is a failure rather than something nobody looks at. Prose elsewhere on this page is ordinary page copy and is not part of that comparison.
What the comparison does not do is judge the words themselves. The description, the reason and the evidence are prose whoever produced the artefact wrote. The build refuses one that states a measurement — an outcome word or a confidence interval standing in the same clause as a number — on a check that did not run, and that refusal is a fixed list of words rather than a reader, so read those columns as description. The columns that carry no result at all for a check that did not run are Compared against, Tolerance and Result.
The honest weakness in that arrangement: the artefact is produced by us, and provenance is a field in it. If you want to check us rather than trust us, the engine source is public, the fixtures are committed, and every expected value cited here comes from a published paper, a government guidance document or a reference implementation you can run yourself. That is the point of publishing the source: the arithmetic is checkable by you, not merely attested by us.