How calibration is scored

Calibration in Hypothesis Lab is mechanical. There's no PM override anywhere in the system — the same input always produces the same number. This page explains exactly how the numbers on your dashboards are computed, so you can trust them.

The scoring logic lives in src/lib/calibration.ts if you want to read the source.

Three things get marked at resolution

When you resolve a hypothesis, you mark each claim with one of four outcomes:

These four marks are the only inputs to every calibration number you'll see.

Prediction accuracy (binary calibration)

The headline number on your dashboard.

prediction accuracy = (HELD + WRONG_MAGNITUDE) / (HELD + WRONG_MAGNITUDE + BROKE)

In other words: a Wrong Magnitude counts as a hit on the binary curve. The reasoning: getting the direction right is the harder, more transferable skill. Magnitude is rated separately.

INCONCLUSIVE claims are excluded from the denominator — they don't help you and they don't hurt you, but they're tracked as a separate rate.

Magnitude accuracy

A second, harder measure: of the claims that were directionally correct, how many were also in the right size band?

magnitude accuracy = HELD / (HELD + WRONG_MAGNITUDE)

A team can have a high binary accuracy and a low magnitude accuracy — they call directions well but miss on sizes. That's a real, distinct skill gap.

Inconclusive rate

inconclusive rate = INCONCLUSIVE / (HELD + WRONG_MAGNITUDE + BROKE + INCONCLUSIVE)

Tracked separately because a high inconclusive rate is itself a signal — usually about how the team scopes hypotheses or sets review dates. The system flags rates above 30% as a warning sign.

Quarter-over-quarter trend

The pill on your dashboard ("↓ 18 pts from last quarter") compares your previous full quarter against the quarter before that. The system needs:

• At least 5 countable claims in each of those two quarters.
• At least 6 months of resolution history total.

Without that volume, the trend is suppressed (no pill shown). Small samples lie too easily.

Per-PM badges (HoP team dashboard)

Every PM gets one of three pills:

• Well calibrated — at least 5 resolved claims, accuracy ≥ floor (50%) and ≥ team median.
• Needs attention — at least 5 resolved claims, but accuracy below either threshold.
• Not enough data — fewer than 5 resolved claims.

The team median is computed across all PMs with at least 5 claims. PMs without enough data are excluded from the median calculation entirely.

High-confidence misses

The "Attention needed" panel surfaces hypotheses that:

• Were rated C4 (High) or C5 (Very high) at creation.
• Were resolved as BROKE.
• Were resolved within the last 90 days.
• Are not private.

These are the most expensive miscalibrations — when a PM was confident and was wrong. The list is sorted newest first.

Confidence calibration ribbon (HoP only)

For each confidence level (C1–C5), the ribbon shows the outcome mix of all the claims the team predicted at that level. A well-calibrated team produces a roughly staircase pattern — C5 mostly green (held), C3 roughly half green half red, C1 mostly green again because very low often means long-shot bet I expect to lose, but worth a try.

If C4 and C5 are mostly red, the team is overconfident. If C1 and C2 are mostly green, the team is underconfident. Both are real, fixable patterns — but only visible at team scale, which is why the ribbon doesn't appear on a PM's personal page.

Why no override?

Calibration scoring being mechanical is the entire point of the product. If a PM could regrade their own bets, calibration would drift toward whatever felt good. The Disagree flag exists so a PM can register their objection — that disagreement is then itself a tracked metric — but it does not change the score.

Previous: Roles and permissions ←    Next: Switching between teams →