What does it mean that measuring systemic change?

Define how you will know the system has actually changed, not just appeared to change. Systemic change is real only when the system produces different outcomes under normal operating conditions — without extra attention, heroic effort, or temporary workarounds. Many change efforts produce initial improvements that fade as the organizational attention moves elsewhere, revealing that the system itself did not change — only the effort level did. Measuring systemic change requires distinguishing between surface changes (different activities within the same system) and structural changes (different system dynamics that produce different outcomes naturally).

Example: An insurance company, Safeguard, implemented a new claims processing system designed to reduce processing time from fourteen days to five. Three months after launch, the average processing time was six days — close to the target. Leadership declared success. Six months later, the average had crept back to eleven days. The system had not actually changed — the initial improvement was produced by extra management attention, temporary additional staffing, and the novelty effect of the new software. When attention shifted and temporary staff departed, the underlying system dynamics reasserted themselves. The measurement mistake was tracking only the outcome (processing time) without tracking the system indicators that would reveal whether the system had structurally changed. A more rigorous measurement framework tracked four indicators: (1) processing time under normal staffing (not temporary surge), (2) processing time when management attention was elsewhere (not during change initiative visibility), (3) error rates (which had actually increased because the new software introduced unfamiliar workflows), and (4) workaround frequency (the number of manual overrides of the automated process, which was high — indicating the system was being fought rather than used). These system indicators revealed that the change was cosmetic, not structural. The team redesigned the implementation based on the system measurements, achieving a genuine five-day average that held under normal conditions.

Try this: For a recent change in your organization, assess whether the system actually changed by applying three tests: (1) The attention test — does the improved outcome persist when leadership attention moves to other priorities? If performance reverts when the spotlight moves, the system did not change — effort changed. (2) The personnel test — would the improved outcome persist if the people who championed the change departed? If performance depends on specific individuals, the system did not change — heroism filled the gap. (3) The stress test — does the improved outcome persist under pressure (high workload, tight deadlines, resource constraints)? If performance reverts under stress, the system did not change — extra capacity masked the underlying dynamics. If the change fails any of these three tests, it is a surface change, not a structural change.