Pre-commit to prediction failure thresholds (X out of Y)…
Set prediction failure thresholds as numeric ratios (X failures out of Y recent predictions) for each schema before observing prediction outcomes to trigger review when the threshold is crossed.
When This Fires
- When setting up monitoring for any operational schema
- During schema documentation when defining review triggers
- When you want to remove subjectivity from the "when to review?" question
- After a schema failure when deciding whether to review or continue
Common Failure Mode
Setting thresholds after seeing the data: "My failure rate is 25%, so I'll set the threshold at 30%." This accommodates the current failure rate rather than challenging it. The threshold should be set based on what failure rate is acceptable for the schema's purpose, not based on what the current rate happens to be.
The Protocol
For each monitored schema: (1) Before observing any outcomes, set a threshold: "I will trigger full review if [X] of the last [Y] predictions fail." (2) A reasonable default: 3 of 10 (30% failure rate triggers review). Adjust based on schema criticality — high-stakes schemas might trigger at 2 of 10. (3) Track predictions and outcomes. When the threshold is crossed → automatic review, not "let me think about whether to review." The pre-commitment is what prevents motivated reasoning from overriding the signal.