Benchmarks
Benchmarks compare METER with simulated ground truth, fitted psychometric models and conventional scores. Most evaluations below were judged against a criterion specified before testing; two were informative without a binary rule and are labelled “No prespecified gate”. The selection carries both the results that passed and the boundaries that did not.
METER uses fixed pretrained weights and is not refitted to each evaluation dataset. Runtime comparisons therefore focus on psychometric models that require dataset-specific estimation. Sum scores are included as accuracy comparators, but not as speed comparators because they are nearly instantaneous.
METER is a measurement platform currently released for research use. It is not a diagnostic, screening or triage system, and no output may be used to make a decision about an individual person.
Can METER do what I need?
| What you want to do | Answer | Evidence |
|---|---|---|
| Score people on a single-scale measure |
| Transfer to three external real datasets (PHQ-9, CES-D-8, EURO-D). |
| Score people on a multi-factor measure, structure supplied (3 to 5 factors) |
| Synthetic confirmation and external transfer. Person scoring only; you declare the structure. |
| Score people on a two-factor measure, structure supplied |
| Returned with a named warning; no confirmatory evaluation of its own. |
| Understand individual items on a single-scale measure |
| Prospective known-truth simulation, criterion set before testing. A research preview. |
| Understand individual items on a multi-factor measure |
| Withheld. One criterion was missed, one quantity was never adjudicated, one was never evaluated. |
| Discover the factor structure |
| Refused by design. The structure is an input you declare, never an output. |
| Compare scores across separate runs or waves | Not offered | Scores from separate runs are not on a common scale. To compare waves or groups, score them together in one run. |
Known-truth simulation
Held-out simulations with known latent traits, scored by the reference module M‑ref
METER: r = 0.939
Fitted GRM: r = 0.944
The reference module approached the performance of a dataset-fitted GRM without being fitted to the evaluation data. M‑ref is an evaluation reference and is not one of the three modules this release serves.
External transfer
66,812 respondents · 28 countries · EURO-D
Agreement with the fitted reference model: r = 0.985
The weights were fixed before evaluation, and METER was applied without refitting to real-world data not used in development.
Multidimensional scoring
20,000 respondents · 50 items · 5 supplied factors
METER: 0.78 s · Confirmatory MIRT: 662.1 s
Median score agreement: r = 0.979
Recorded runtime was approximately 850 times lower than confirmatory MIRT in this evaluation.
Selected benchmarks
These selected benchmarks include successful results and important capability boundaries. Full methods and evaluation results are available in the preprint and supporting evidence record.
| Evaluation | Instrument | Evaluation size | METER vs reference | Target-data refitting | Recorded runtime (reference vs METER) | Evaluation verdict | Capability status |
|---|---|---|---|---|---|---|---|
| Synthetic 1-D recovery, 40 unseen worlds | synthetic GRM worlds (module M-ref) | 17,285 | person r 0.939 vs generating trait (mirt 0.944, raw sum 0.923) | No | 3.9 s/world fitted vs 0.011 s/world pretrained | PASS | — |
| NHANES 2017–18, locked zero-shot | PHQ-9 | 5,533 | 0.9371 vs MML-GRM EAP | No | 9.7 s vs 0.12 s | No prespecified gate | SUPPORTED |
| ESS Round 7, 21 countries | CES-D-8 | 40,185 | pooled 0.9696; per-country 0.9545–0.9899 | No | 11.2 s vs 0.56 s | No prespecified gate | SUPPORTED |
| SHARE wave 9, locked, 28 countries (pre-registered gate) | EURO-D | 66,812 | 0.9851 [0.9849–0.9854]; 0.967–0.990 per country | No | 13.8 s vs 6.4 s | PASS | — |
| SHARE wave 8, locked temporal replication | EURO-D | 51,732 | 0.9847; 27/27 countries 0.961–0.990 | No | 7.4 s vs 2.5 s | PASS | — |
| IPIP Big-Five markers, locked, K = 5 | IPIP-FFM (module M-struct) | 20,000 | median 0.9791; per-factor 0.960–0.984 vs confirmatory MIRT | No | 662.1 s vs 0.78 s | PASS | — |
| Longitudinal prospective noninferiority vs raw-sum mean (ELSA; SHARE-B), locked | CES-D-8; EURO-D | 10,805; 20,132 | deficit lower bounds −0.0237 / −0.0236 vs −0.02 margin | No | — | FAIL ×2 (preserved boundary) | — |
| Reliability indicator: conditional coverage (P5, truth-referenced) | 400 held-out synthetic worlds; 12 prespecified design strata | 400 worlds | coverage 0.785–0.983; 10 of 12 strata outside the [0.93, 0.97] bandFailed the prespecified band with direction differing by response format (binary too narrow; 5- and 7-category too wide), so one global scalar calibration is not supported across those strata; the uncertainty output is therefore released as a relative reliability indicator (ordinal use only). | No | — | FAIL | — |
Legend. Evaluation verdict reports whether a prespecified evaluation rule was passed or failed. “No prespecified gate” identifies evaluations that were informative but had no binary pass/fail rule. Capability status is assigned separately under the capability contract; “—” means that no separate row-level capability designation applies. Unless otherwise stated, r is Pearson correlation: agreement between person scores on real data and recovery against the generating latent trait in simulations. Spearman correlations are explicitly labelled ρ.
What METER currently supports
Support depends on the requested output and study design. The table shows which inferences METER currently supports, limits, has not yet established, or does not offer—and the evidence behind each decision.
Evidence for person scores includes transfer to external real-world datasets. Evidence for item parameters currently comes from prospective simulations with known generating values.
Public capability contract v3.7 · frozen scoring contract v3.6, unchanged · claim rules claims‑v1
- SUPPORTED
- A criterion set before testing was met by evidence of the stated kind. An estimate is returned. It is not a claim of superiority over conventional psychometric models.
- LIMITED
- Returned, with a named warning, but without a confirmatory evaluation of its own. An estimate is returned.
- NOT CONFIRMED
- A criterion set before testing was evaluated and not met. The estimate is withheld. This is the strongest negative here: something was measured and fell short.
- NOT ESTABLISHED
- No adjudicated evidence exists for the claim. The estimate is withheld. A quantity may still have been computed and reported descriptively, as two-factor discrimination was; what is missing is a criterion it was ever judged against.
- NOT TESTED
- Never evaluated in this evidence generation. The estimate is withheld as untested rather than as failed.
- NOT SUPPORTED
- Outside every METER capability contract to date. Refused by design, for every dataset.
The four statuses that withhold return no number at all. METER declines the request and says why, rather than returning a value the evidence does not stand behind. The same vocabulary is served in machine-readable form at /api/capabilities.
v3.7 is additive. It carries every v3.6 decision forward unaltered and adds only the single-factor item rows below, which rest on a later evidence generation. Evidence from one model generation never raises a claim served by another.
| Claim | Stratum | Status | Evidence source |
|---|---|---|---|
| Person scores | K = 1 (single scale) | SUPPORTED | External real-data transfer without refitting (PHQ‑9, CES‑D‑8, EURO‑D; see the table above) |
| Person scores | K = 3–5, supplied structure | SUPPORTED | Confirmatory synthetic benchmark plus one external transfer (IPIP, artifact D9E) |
| Person scores | K = 2, supplied structure | LIMITED | Expressible and returned with a named warning; no confirmatory evaluation of its own (the confirmatory benchmark established 3–5 factors). B4X‑3 criterion C1 passed on all three seeds, but it is a pooled K ≤ 2 person-score result from a different model generation. No K=2-only person-score criterion was defined, panelled or adjudicated, so C1 does not raise this row. |
| Item discrimination | K = 1 | SUPPORTED · research preview | B4X‑3 C2: prospective synthetic known-truth confirmation, PASS on all three seeds (medians 0.922 / 0.909 / 0.929 vs line 0.865) |
| Response thresholds | K = 1 | SUPPORTED · research preview | B4X‑3 C3: prospective synthetic known-truth confirmation, PASS on all three seeds (RMSE medians 0.459 / 0.377 / 0.343 vs line 0.500) |
| Response thresholds | K = 2 | NOT CONFIRMED | Not confirmed under the prespecified rule (B4X3_NOT_CONFIRMED). B4X‑3 C4 cleared its line on the across-seed median and on two of three seeds; seed 20260901 missed by 0.001794, and the rule has no “almost passed” branch (details below) |
| Item discrimination / loadings | K ≥ 2 | NOT ESTABLISHED | Outside the B4X‑3 designation: computed and reported descriptively, never adjudicated; earlier K=2 loading inference closed as a capability boundary |
| Item parameters | K ≥ 3 | NOT TESTED | The B4X‑3 evaluation corpus admitted only worlds with two factors or fewer, so nothing at this dimensionality was evaluated on any estimand. Withheld as untested rather than as failed |
| Latent-structure discovery | any | NOT SUPPORTEDcapability status; a request returns REFUSED | Listed separately from factor-number discovery so neither can be read as permitted by the other’s absence: METER is told which items belong to which factor |
| Factor discovery | any | NOT SUPPORTEDcapability status; a request returns REFUSED | Never offered: structure is an input you declare, not an output (Milestone 7 closure) |
Technical validation details
Broad-prior prospective confirmation (B4X-3)
Three model seeds were evaluated on a held-out synthetic corpus using four criteria specified in advance. Confirmation required every criterion to pass both on the across-seed median and on each individual seed. The corpus was sealed: the evaluation grant did not exist while the model was being trained.
| Criterion | Stratum | Per-seed medians | Pre-registered line | Per-seed verdicts |
|---|---|---|---|---|
| C1 · person scoring (theta r) | K ≤ 2 | 0.8388 / 0.8392 / 0.8399 | ≥ 0.7990 | PASS · PASS · PASS |
| C2 · item discrimination (r) | K = 1 | 0.9224 / 0.9088 / 0.9293 | ≥ 0.8646 | PASS · PASS · PASS |
| C3 · thresholds (RMSE, logit) | K = 1 | 0.4590 / 0.3767 / 0.3429 | ≤ 0.5002 | PASS · PASS · PASS |
| C4 · thresholds (RMSE, logit) | K = 2 | 0.4504 / 0.4210 / 0.3987 | ≤ 0.4486 | FAIL · PASS · PASSAcross-seed median passes (0.4210) and two of three seeds pass; seed 20260901 fails its line by 0.001794. Under the pre-registered rule that is a FAIL. |
Overall result: not confirmed under the prespecified rule (B4X3_NOT_CONFIRMED).
Three of four prespecified constituent capabilities passed on every seed. K=2 threshold recovery did not satisfy the prespecified every-seed rule; therefore the overall Scope-B designation was not confirmed. No post-result rescue was performed.
The K=1 item-discrimination and response-threshold criteria nevertheless passed for all three seeds. METER therefore reports these K=1 outputs as research-preview capabilities while withholding multidimensional item-parameter outputs that have not been established.
K=2 item discrimination was evaluated descriptively and did not contribute to the overall verdict. Its per-seed medians were 0.787 / 0.761 / 0.811 against a historical 0.842115 line; under the frozen rule that is evidence about corpus difficulty, not about METER’s capability, in either direction, and it could neither rescue nor veto the designation. The frozen closeout’s one-sentence summary of where these medians sit relative to a historical development band was corrected by an append-only erratum (B4X3_ERRATUM_001, in the evidence record); the values and every capability decision are unchanged.
What “passed” means here. Each line is a non-inferiority bar against a conventional estimator refitted on every world in the corpus, not an absolute quality bar. Passing means METER came within the pre-registered margin of that per-world fit without being fitted itself. It is not a finding that METER is more accurate than conventional psychometric models; the claim is inference across many different measures at low cost.
Two constraints on how this result may be restated. C1 is a pooled K ≤ 2 person-score result and is not a separately adjudicated K=2-only person-scoring capability; no such criterion was defined, panelled or adjudicated. And anything at K ≥ 3 was not tested in this evaluation on any estimand, so it is withheld as untested rather than as failed.
How to interpret these results
Agreement is not accuracy. On real data, agreement with a reference estimator does not establish that either method recovered the true latent trait. The methods may share assumptions and errors. Direct recovery against known truth can be assessed only in simulations. In METER’s simulations, agreement between estimators near 0.990 sometimes coexisted with truth-recovery correlations of 0.79–0.83.
Temporal replication is not independent replication. The SHARE wave 8 analysis repeated the evaluation at a different time point, but 38,071 participants also appeared in wave 9. It is therefore a temporal replication within an overlapping longitudinal panel, not an independent-cohort replication.
View technical provenance
Evaluation record. The numbers on this page are transcribed by hand from two frozen records and are not read at build time. The selected-benchmarks table comes from the versioned evidence matrix at repository tag METER_FOUNDATIONAL_EVIDENCE_FREEZE; the broad-prior section and the capability table come from reports/b4x/b4x3_evaluation.json at commit 2d12386, closed at ea1dd8b, which postdates that tag. Every capability status in the table below, its row count, and the B4X-3 per-seed medians, criterion lines and shortfall are checked against the release ledger by the test suite, which fails if one drifts. The selected-benchmarks rows above, and the prose on this page, are transcribed and reviewed by hand. Rows are never edited on this page; corrections happen upstream in the frozen records.
Which checkpoint serves. Item parameters are produced by A-FREE-EQ_20260902, the fixed serving checkpoint chosen for reproducible deployment. It was not selected for performance and no cross-seed selection took place: per-seed checkpoints were chosen inside the training loop, before the evaluation corpus existed. The capability rests on the single-factor criteria passing on all three seeds, not on this checkpoint’s own results.
Outcome vocabulary. PASS denotes a pre-registered gate that was met, and FAIL a pre-registered gate that was not; “No prespecified gate” identifies an evaluation that was informative but had no binary pass/fail rule. The capability-status column records the status such an evaluation established, and “—” means no separate row-level capability designation applies. Both are evaluation-study outcomes and are separate from the capability decision a user receives for their own run.
Notes. Raw sums remain strong for some simple instruments (ESS7 sum-agreement 0.998; the longitudinal raw-sum mean beat the prespecified noninferiority margin twice); METER is not claimed to universally outperform raw totals. Runtime ratios labelled “recorded” are explicit artifact fields; the largest ratios in the repository (D9C ~12,000x, D9B ~5,000x) come from NOT_EVALUABLE and FAIL runs and are deliberately excluded from the headline cards. The D9C and D9E artifacts both record comparator fit 662.1 s; the ~850x figure survives on the D9E numbers alone.
Module attribution. METER is a release of separately trained, separately hash-pinned modules. The synthetic recovery row was produced by the 1-D reference core (M-ref). All real-data unidimensional rows come from the scoring core (M-score, 42,899 parameters, state 224a701b…). The IPIP row comes from the known-structure head (M-struct, state 8d427520…). The IPIP pass is artifact D9E.
Methods and full evaluation details are available in the preprint. Claim-level evidence and limitations are documented in the model and capability card.