What is isolate variables when optimizing?

Isolate variables when optimizing is a concept in personal epistemology: Change one thing at a time so you can attribute improvements to specific changes.

Example: You run an AI agent that summarizes customer support tickets and routes them to the right team. Response accuracy is 72 percent, and you want to reach 90. You have three ideas: rewrite the system prompt with clearer routing criteria, switch from GPT-4o to Claude, and add a retrieval step that pulls the five most similar resolved tickets as context. You implement all three changes at once. Accuracy jumps to 88 percent. Progress — but which change drove the improvement? Maybe the new prompt alone would have reached 88 percent and the model switch added nothing. Maybe the retrieval step actually hurt accuracy by injecting irrelevant examples, and the gains came entirely from the prompt and model changes. Maybe the model switch introduced a regression that the retrieval step masked. You cannot know, because you changed three variables simultaneously. Now imagine you made one change per week. Week one: rewrite the prompt. Accuracy moves from 72 to 81 percent. You now know the prompt rewrite is worth nine points. Week two: switch the model. Accuracy drops from 81 to 78 percent. You now know this model performs worse with your prompt — revert the change, save the API cost. Week three: add the retrieval step. Accuracy moves from 81 to 91 percent. You exceeded your target, and you know exactly which component delivered each increment. More importantly, you avoided shipping a model change that actually degraded performance — a regression you would never have detected if you had changed everything at once.

This concept is part of Phase 29 (Agent Optimization) in the How to Think curriculum, which builds the epistemic infrastructure for agent optimization.