criteria.yaml

criteria.yaml is the customer-authored half of a run. It carries the declarations the engine refuses to infer — positive class, score orientation, thresholds, prevalence, subgroups — and, optionally, the acceptance criteria you have set. The engine validates the file against schema/criteria_schema.json and supplies no default for anything a manufacturer must own. It is specified (D1) to echo the file verbatim into the T8 run manifest; the manifest writer is not in engine 0.1.0.dev1.

Engine status. In 0.1.0.dev1, proofpack run loads and validates this file (io/declare.py), runs the HALT gates, coarsens a declared period column, and stops after ingest. Every block and field below is read and checked today, and every HALT named below fires today. Wherever this page says what the pack reports, records, renders, echoes or labels — the met / not met / not assessable reading, the attainable-lower-bound note, the optimism flag, PPV/NPV at declared prevalence, calibration, PPA/NPA wording, the both-ways tables, the cluster bootstrap, the exploratory label, the T8 manifest, the ledger — it is describing D1’s specification for a later engine build day, not code in 0.1.0.dev1. The interval and bootstrap library code exists under stats/. run today computes the AUROC point estimate gate H01 tests (auroc_mann_whitney, the one name gates.py imports from stats/; the value is kept in memory and written to no file — it reaches the terminal only when H01 halts, as auroc_declared_orientation in that HALT’s detail) and the observed prevalence gate H10 compares with each declared prevalence value (written to ingest_report.json, rounded to four places, inside the W10 finding’s detail when the flag fires). Neither cli.py nor gates.py contains the strings bootstrap, delong, wald_ci, logit_ci, auroc_number, subgroup or calibrat; tests/day5-repair2.test.mjs greps both files for those seven strings when the engine checkout is reachable.

Two rules from the engine’s io/declare.py decide whether a run starts:

• a missing mandatory block, or a criterion or fairness block with an empty author, date or justification, is HALT H08;
• a criterion that names an unknown metric, operating point, attribute or level is HALT H09.

The file allows no property the schema does not list (additionalProperties: false), so a misspelt key is refused rather than ignored.

Declarations

These are the blocks the engine reads before it computes anything. They are declared, never inferred: ProofPack does not look at your data to guess which label means “condition present”, which way the score runs, or where the threshold sits. Every value in this section is authored by you; ProofPack sets no default.

schema_version

Required. The integer 1.

model

Required. name (required, non-empty) and version (required; a string or a number). Optional prior_version (string, number or null) for version comparison and udi_di (string or null).

task

Required. One of binary, multiclass, regression, segmentation_ingest. Version 1 computes the full pack for binary; the other three are the descriptive lite tasks.

classes — mandatory

positive and negative, each a string or an integer: the label values in y_true that mean target condition present and absent. They may not be equal. Any y_true value outside these two plus the declared indeterminate values is HALT H02. Authored by you; ProofPack sets no default.

score — mandatory

• type: probability, logit or other. Calibration is computed only for probability, and a probability outside [0, 1] is HALT H03.
• orientation: higher_is_positive or lower_is_positive. If the declared orientation contradicts the data (AUROC below 0.5) the run is HALT H01.

Authored by you; ProofPack sets no default.

operating_points — mandatory, at least one

A list. Each entry:

• id (required, non-empty; ids must be unique across the list);
• threshold (required, a number);
• rule (required): >=, >, <= or < — the comparison of score against threshold that yields a positive prediction;
• provenance (required): prespecified_sap, derived_from_this_dataset or other. A threshold derived from this dataset is specified to be reported with the optimism flag beside every metric computed at it (not in engine 0.1.0.dev1, which computes no metric yet);
• source (optional): where the threshold was set, echoed in T8.

If the table also carries y_pred, it must equal the rule applied to score at every declared operating point on every row, else HALT H04. Authored by you; ProofPack sets no default.

reference_standard — required

• type: reference_standard or comparator. With comparator the pack reports positive and negative percent agreement (PPA/NPA) and suppresses the words sensitivity and specificity, following FDA’s 2007 statistical guidance on diagnostic tests (final);
• description (optional): your text, echoed.

indeterminates — required

• policy: report_both_ways or none_present. Both-ways renders the as-positive and the as-negative table;
• values (optional): the label values that mean indeterminate. None of them may coincide with a declared class.

clustering — required

• unit: none or case_id. With case_id, AUROC and proportion intervals use the cluster bootstrap, and the analytic methods (DeLong, Wilson) are refused rather than silently used;
• declared_by (optional): who decided.

With unit: none, duplicate case_id rows that disagree on y_true are HALT H05.

prevalence — mandatory, at least one entry

A list of {label, value, source}: label (required, non-empty), value (required, strictly between 0 and 1), source (optional). PPV and NPV are specified to be computed at each declared prevalence (not in engine 0.1.0.dev1). If the observed prevalence differs from a declared one by more than 0.10 absolute the run continues with flag H10, printed as finding W10 and exit code 2 — that part runs today. Authored by you; ProofPack sets no default.

subgroups — mandatory

A list of attribute declarations. Each entry:

• attribute (required): a canonical column name matching ^[a-z][a-z0-9_]{0,31}$ — one of the schema’s attribute columns or an attr_* column;
• prespecified (required, boolean). An attribute you tabulate without prespecifying it is labelled exploratory;
• source (optional);
• reference_level (optional; string, integer or number): the level the others are compared against;
• bands (optional): a list of [low, high] pairs for a numeric attribute. A numeric age column with no bands declared is HALT H08 — there are no default bands.

Attributes must be unique. A declared attribute or level the table does not carry is HALT H09. Authored by you; ProofPack sets no default.

period

Optional, but required the moment the table carries a date-like column: column (the column’s name) and granularity (quarter, month or year). Dates are coarsened to that granularity at ingest and never appear in any output. A date-like column with no period declaration is HALT H11.

Acceptance criteria — criteria

Optional. If the block is absent the pack is specified to report estimates and intervals only, with no met / not met column anywhere. If it is present, the engine is specified to report each criterion as exactly one of criterion met, criterion not met or not assessable (with a typed reason), and nothing else: ProofPack never chooses a margin and says nothing about the model beyond those three words. That evaluator is not in engine 0.1.0.dev1: today each entry is validated (H08, H09) and nothing is computed from it.

Each entry (all of it authored by you; ProofPack sets no default):

• id (required, non-empty);
• metric (required): one of the metric ids below. The schema accepts any non-empty string here and io.declare checks it against the enum, so an unknown id is HALT H09, not a schema error;
• type (optional): point (the default reading) or paired_difference_vs_prior — a version comparison criterion against your own margin;
• operating_point (optional): the id of a declared operating point; an unknown id is HALT H09;
• scope (optional): the string overall, or {attribute, level} — a named level, or "*" for every level of that attribute;
• statistic (required): ci_lower_bound or point_estimate;
• comparator (required): >=, >, <= or <;
• value (required): a number — your threshold or margin;
• author, date (YYYY-MM-DD) and justification (all required and non-empty; empty is HALT H08).

For every ci_lower_bound criterion the engine is specified to report the largest lower bound attainable at the observed sample size and to mark “criterion not attainable at observed n” where that falls below value, without offering a sample size. Not in engine 0.1.0.dev1.

Metric ids

sensitivity, specificity, ppa, npa, ppv, npv, accuracy, balanced_accuracy, f1, mcc, lr_pos, lr_neg, dor, youden, auroc, auprc, brier, ipa, oe, calibration_slope, calibration_intercept, ece, tpr_gap, fpr_gap, ppv_gap, npv_gap, auroc_gap, psi, ks_d, prevalence, kappa, macro_auroc, bias, loa_low, loa_high, mae, rmse, ccc, dice_mean, dice_median.

fairness

Optional. Exactly one criterion of interest, because no set of fairness criteria can all hold at once:

• criterion_of_interest (required): tpr_gap, fpr_gap, ppv_gap, npv_gap, auroc_gap or calibration_by_group;
• attribute (required);
• bound (optional; number or null): a numeric bound yields met / not met; absent, the gap is descriptive only;
• author, date, justification (required, non-empty).

Authored by you; ProofPack sets no default.

ledger

Optional. warn_after_acceptance_runs (integer, at least 1): how many runs against acceptance criteria on the same test set the local ledger is specified to tolerate before it warns. Customer-set; no default exists. The ledger is not in engine 0.1.0.dev1 (see version comparison).

egress

Optional. telemetry (boolean) switches the single telemetry call off; --offline does the same for a run. suppression sets the k-suppression thresholds applied before anything is serialised for egress: min_n (minimum 10), min_events (minimum 5), min_nonevents (minimum 5).

This is one of the two blocks that does carry engine defaults — 10, 5 and 5, recorded in the egress schema — and you may only make them stricter; the schema refuses a looser value.

bootstrap

Optional, and the other block with engine defaults: B (integer, at least 1; default 2000), seed (integer, at least 0; default 20240101 — numpy refuses a negative seed, and the schema refuses it here rather than half way through a run), interval (percentile only; BCa is deferred and declaring it fails validation rather than silently yielding a percentile interval). The seed and B are specified to be recorded in the manifest together with whether they were declared or defaulted (the manifest is not in engine 0.1.0.dev1).

Required at the top level

schema_version, model, task, classes, score, operating_points, reference_standard, indeterminates, clustering, prevalence, subgroups. Everything else is optional; nothing outside the list above is allowed.

A scaffold, not a default

proofpack declare --out criteria.yaml is specified to write an interactive scaffold that refuses to save without author, date and justification and pre-fills no values. It is not in engine 0.1.0.dev1; write the file by hand from this page until it lands.