Version comparison

proofpack compare takes the new version’s table and the prior version’s table, scored on the same test set, and reports the paired difference for every metric with a confidence interval. A criterion of type paired_difference_vs_prior in criteria.yaml turns one of those differences into criterion met / not met against your margin — the engine ships no margin and never suggests one. This is the pack a predetermined change control plan’s performance-evaluation report is built from; the PCCP guidance is final (December 2024, updated August 2025).

proofpack compare --input new.csv --prior old.csv --criteria criteria.yaml \
  --mapping mapping.json --out ./pack --offline

Pairing

Both tables must carry row_id, and the two sets of ids must be identical; each row of the new table is paired with the row of the same id in the prior table. A mismatch is HALT H12.

--allow-unpaired turns the halt into finding W12: the comparison proceeds with unpaired methods and every difference is labelled “not like-for-like” in the pack, because two versions scored on different samples are not a before-and-after of one change. The label is not removable.

Both tables pass through the same ingest and the same HALT gates as a run, against the same declarations. The exit code is 2 if either table or the pairing produced a finding, else 0.

What engine 0.1.0.dev1 does

Paired ingest of both tables and gate H12, writing compare_ingest_report.json into --out. The paired statistics exist as library code in the engine — the paired AUROC difference with DeLong’s method (refused on clustered data, where the cluster bootstrap is used instead) and the Newcombe paired interval on a proportion difference — but the compare command does not yet call them, and the T2 document is not yet rendered; that is the comparison build day’s work, listed on the validation page as pending until its fixtures pass.

Test-set ledger

Every run against acceptance criteria on a given test set is one look at that set. ProofPack is specified to keep a local ledger — a file in your environment, never sent anywhere — of how many acceptance runs each header-set hash has seen, and to warn once the count passes the ledger.warn_after_acceptance_runs you declared. There is no default: you set the number, and the warning is recorded in the T8 manifest so a reviewer can see it.

The ledger records looks; it does not stop them and it does not adjust any interval. Re-rendering a pack, comparing versions or monitoring drift are not counted as acceptance runs; only a run whose criteria block is evaluated is.

Engine 0.1.0.dev1 validates the ledger declaration and does not yet write the ledger file; the counting lands on the engine day that evaluates criteria.

Long format

The alternative to two files is one file with a model_version column and rows for both versions; row_id still pairs them. Declare model.version and model.prior_version so the pack labels the two arms.