Skip to content
How to ThinkIn the Age of AI

Question

What is comparing agent versions?

beginner

Quick Answer

Run two versions of an agent simultaneously and let the data tell you which performs better.

Example: You have a personal planning agent that generates your weekly priorities. It works — but you suspect it over-weights urgency and under-weights importance. So you build a second version with a modified prompt that explicitly ranks tasks by long-term impact before considering deadlines. For three weeks, you run both versions every Sunday evening. Version A produces your familiar urgency-biased list. Version B produces an importance-weighted list. You do not decide in advance which is better. You track three metrics: how many priorities you actually complete, how satisfied you are with the week's output on Friday, and how often you override the list mid-week. After three weeks, Version B wins on all three metrics — higher completion, higher satisfaction, fewer overrides. You did not improve your agent through intuition. You improved it through comparison.

This concept is part of Agent Optimization in the How to Think curriculum.

Learn more in these lessons