Check subgroup relationships before aggregating…
Before aggregating data across subgroups, check whether the relationship holds within each subgroup independently, as aggregate patterns can reverse at the disaggregated level (Simpson's paradox).
When This Fires
- Before making any decision based on aggregated performance data across teams, departments, or categories
- When comparing overall rates between two entities that differ in their subgroup composition
- During data analysis when combining results across demographic, geographic, or categorical subgroups
- When someone presents a compelling aggregate statistic that will drive a significant decision
Common Failure Mode
Trusting aggregate statistics because "the sample is large enough." Sample size is irrelevant to Simpson's paradox — the reversal can be perfectly statistically significant at both the aggregate and subgroup levels. The issue isn't noise; it's confounding. A large sample doesn't protect you from a lurking variable that changes subgroup proportions.
The Protocol
(1) Before acting on any aggregated comparison, ask: "Are there relevant subgroups that differ in size or baseline characteristics?" (2) If yes, disaggregate: compute the relationship within each subgroup independently. (3) Compare: does the relationship hold in the same direction within each subgroup as in the aggregate? (4) If the direction reverses in any subgroup → the aggregate is misleading. Report and act on the subgroup-level analysis. (5) If the direction holds consistently across subgroups → the aggregate is reliable for that specific comparison. (6) When in doubt about relevant subgroups, try the obvious ones: size categories, severity levels, time periods, demographic groups.