Mereth Labs
The standard of evidence

The standard of evidence

Mereth Labs grades its own work against the rubric on this page. It is published so that the lab’s progress reports can be checked against fixed goalposts, and versioned so that any change to the goalposts is visible.

Version 1 · July 2026. Revisions are dated and explained.

What we measure

Every project names its metrics before it runs. The primary outcome across the lab is learning gain per active minute against a matched baseline: what a learner gained, per minute of real effort, compared with the best credible alternative — not compared with nothing.

Six further quantities are reported alongside it. No single metric is treated as sufficient.

  • Delayed retention at fixed horizons — a week, a month — rather than exit-quiz performance.
  • Transfer to tasks the system never drilled.
  • Calibration of the learner model: at a stated 80% mastery, observed success should be near 80%.
  • Feedback latency, reported as p50 and p95 against a stated budget.
  • Robustness to gaming, for the learner and for the institution holding the incentive.
  • Equity across subgroups: a gain that appears only for already-advantaged learners is recorded as a failure.

One further test applies to every system: the removal test. Performance is measured after AI assistance is withdrawn. In a 2024 randomised trial, students with unstructured chatbot access scored 48% better during practice and 17% worse on the exam without it. A system whose gains disappear with the model was doing the learner’s work, not building the learner’s skill.

The four levels

Every piece of work sits at exactly one level. Claims are made at the level the work has earned.

L0 — Argued

A mechanism grounded in prior results, written as a falsifiable brief: the learner-state variable, the observable signal, the intervention policy, the evaluation metric, and the result that would disprove it. All projects enter here. Nothing ships from here.

Passes when the claim is stated precisely enough that a disconfirming test could be designed from the brief alone.

L1 — Instrumented

The loop runs end to end and logs the quantities named in the brief: item-level traces, latency percentiles, learner-model calibration curves. At L1 the system can be wrong in specific, inspectable ways.

Passes when every quantity in the L0 brief is measured in the running system, and a week of logs is sufficient to reproduce any chart the lab publishes.

L2 — Validated

The loop outperforms a credible baseline on pre-registered outcomes with real learners. Outcomes and analysis are fixed before data collection. The baseline is the best available alternative. Engagement metrics are not admissible as success criteria.

Passes when the pre-registered primary outcome improves, with uncertainty reported, in a study that someone outside the lab could re-run.

L3 — Durable

The gain persists under delayed retention, transfer, gaming pressure and subgroup analysis, and survives the removal test. L3 is the only level at which the lab describes a system as working, without qualification.

Reporting rules

Effect sizes are reported with uncertainty, not as bare percentages. Negative results are published with the same care as positive ones. Cross-study comparisons carry their caveats: a 0.3σ result from a Nigerian after-school programme and a 1.3σ result from a Harvard physics course are both real and are not the same thing. Instruments and evaluation code are released with the papers where data privacy allows. A metric used as a success criterion is never used as a learner-facing reward, to keep the measurement independent of the incentive.

What would change our conclusions

The lab’s load-bearing assumptions, and the observations that would overturn them:

  • The loop is the bottleneck. If well-instrumented loops repeatedly fail to beat good baselines outside lab conditions, the binding constraint is elsewhere, and the research programme changes.
  • AI can preserve the mechanism. If removal-test deficits keep appearing in systems that follow this standard, then current models substitute for thinking rather than scaffold it, and the product programme changes.
  • Measurement survives incentives. If the lab’s own metrics inflate the way test scores historically have — score up, competence flat — the standard has failed and is revised from that record.

Retired assumptions are documented in notes.