What Happens When Agents Can Teach Your Methodology Better Than You Can

Andrej Karpathy — one of the people who built the AI systems we’re all now using - recently shared something that stuck with me.

He built a system where an AI agent runs experiments autonomously overnight. There’s no human involvement, the human doesn’t even direct the individual experiments.

All the person does is write a plain English file that tells the agent how to research - then the agent loops, tries something, measures the result, keeps or discards, then repeats.

100 experiments done while you sleep.

The thing that makes it work is a clean score - The agent can judge ‘does this work?’ because the answer is a number - the lower that number is, the better it worked. No ambiguity at all.

When we think about coaching, it doesn’t have a clean score - the feedback is qualitative, subjective and also very slow to produce. You can’t run 100 coaching experiments overnight and rank them, there’s no number you can get.

The human is the scoring function.

So we’re the bottleneck, but also the moat.

When you try and add your methodology to agents - your sequencing, decision trees, the ‘If I say this, explore this’ logic - you can get good outputs.

But the agent can’t tell whether it’s output was good - whereas I can.

That human judgement is built on years of pattern recognition, but also on hardware that evolved for connection. It can’t be automated yet.

But I can’t say it won’t ever be - the direction of travel is clear, and whoever figures out ways to score qualitative work in whatever domain, will reliably turn it into an autonomous loop that could be optimised.

But right now, the question for coaches isn’t ‘Will AI replace me?’, It’s more interesting than that.

It’s this - If you tried to write the scoring criteria for your own work - the rules for what makes your coaching good — how specific could you get?

The coaches who can answer that in detail are the ones whose IP is genuinely theirs. The ones who can’t, probably don’t have much that’s different from everyone else.