After a very finance-oriented previous Leaders newsletter, today, we’re putting ourselves at the bleeding edge of this technology.

For that, we will cover research that answers two questions:

By the end of this read, you’ll have a better understanding of this technology than you ever thought you would have. If AI is about to transform our future, understanding it better than anyone else around us feels like a key competitive advantage.

Let’s dive in.

In frontier research today, the hottest topic right now is on-policy self-distillation, or OPSD, the four hottest words in San Francisco right now.

I know, it sounds complicated because it’s named to sound like it, but it’s actually not at all if you break it into first principles, which is what this newsletter is all about.

And to break down frontier research into first principles, we need to start humbly.

Simplified much, all AI models follow the exact same process to learn: they make a prediction about what we want to learn, we measure that prediction against the ground truth (what it should have predicted), and we use the difference between the two predictions as the learning signal.

If the AI should have predicted 10 and it predicted 1,000, the difference is 900, a large error. If the next prediction is 500, the difference is lower, prompting the model to continue heading in that direction.

However, AIs never predict single numbers (or rarely); they almost always predict probability distributions. In other words, they don’t say “the answer is 10,” but “I believe the answer is 10 with 90% certainty, but it could be 12 with a certainty of 10%.”

This means the AI has responded with two numbers, not one, and associated probabilities, creating a distribution of possible responses.

Notice the word “believe,” because I used it on purpose. By outputting a distribution, the model expresses its degree of uncertainty about its prediction; we are still forcing it to make a choice, but we allow it to indicate how “convinced” it is about the prediction.

Distributions are great because they prevent “response collapse”: if you force the model to always choose one and only one option, creativity goes out the window because the model learns only the most likely option, not the others.

That’s the difference between a Large Language Model (LLM) that always responds “Let’s bake a cheesecake” to the question “What shall we bake today?” and one that responds with a more varied set of possible pastries to bake.

It’s important that you understand this concept, so keep it in mind for a few minutes until we get into the weeds of OPSD.

Ok, so models output distributions, and these are compared to the actual response, fine. However, the way we apply this guidance determines how the model learns.

But what do I mean by that?

Specifically, we give the model a sequence of, say, three words, and we hide the fourth. The model outputs what it believes is the fourth word, and we then compare it to the actual fourth word.

“Comparing” here means looking at the probability it assigned to the correct word. For example, say the sequence is “What’s the capital of Mongolia?” The model then assigns a probability to all words it knows—in reality, it outputs ‘tokens’, which can be syllables or entire words, but let’s treat them as words nonetheless—but we only care about the probability it assigned to the right one.

In this case, we look at the probability it assigned to Ulaanbaatar, which is, say, 68%. As a perfect model would have assigned 100% probability to that city, the 32% gap serves as a learning signal indicating how wrong the model was.

Then, the model is updated so that next time it sees those words, that prefix, it’s more likely to assign a higher probability to ‘Ulaanbaatar’. Over trillions and trillions of predictions, eventually it will consistently assign the highest probability to the correct words.

Training an LLM like Claude or ChatGPT is doing what we’re describing, but in two ways: imitation and reinforcement.

Imitation learning is when we give the AI the entire sequence of words it needs to learn. The key thing about imitation learning is that every single word is a learning opportunity because we have “full supervision”: for every prediction, we measure how well it matches the ground truth.

This sounds ideal because the model is offered a dense learning opportunity; every prediction is a learning signal. It’s like having a genie on your shoulder that gives you feedback for every decision you make in your life. That is why it’s called imitation: the model is simply tasked with imitating given sequences.

The process is as follows: We give the model the entire sequence. For every word in the sequence, we ‘mask’ the future words, meaning it can only see the words up to that point, and ask: What is the next one?

As we do this for every word in a sequence, if the sequence has 12 words, the model makes 11 predictions; 11 learning opportunities.

For the sequence “Julius Caesar crossed the Rubicon in 49 BC”, the model will make 7 predictions:

In other words, the AI model outputs all predictions in a single pass (called a ‘forward pass’ in AI parlance) because for each prediction, we just mask the future words.

However, at this point, you can already guess what the implication of only using imitation learning is: the model is tempted to just learn whatever is given; it’s tempted to just memorize.

Think of this as a student in a maths class who “learns” solely by imitating problem solutions. Are they really learning, or just memorizing?

That is why, just like humans, AIs go through an extra layer of learning we call Reinforcement Learning, but it’s basically a cool term for trial and error.

In this phase, the model is given a question but not the answer. Therefore, the model needs to start generating the answer without feedback until it thinks it has it. Once the final solution is there, we then compare that word, only that one, to the actual answer, resulting in correct or incorrect feedback, but without revealing the process to get to the correct response.

That’s why it is no longer imitation; the model has to figure it out on its own. Once the model solves the problem (reaches the correct response), we update it, assuming that something in its “reasoning” must have gone right for it to reach the correct response, thereby reinforcing the behaviors (hence the name of the technique) that occurred during that process.

However, there are two big issues with Reinforcement Learning: verifiability and exploration complexity, the final two concepts one must understand to see how important on-policy self-distillation is today.

It’s important to acknowledge that not all domains are verifiable, meaning there are many areas, like writing or art, where there’s no universally accepted definition of greatness or even correctness.

What makes a Da Vinci painting great? There’s a lot of attribution value by name, of course, but what makes the Mona Lisa a superior painting to a Rubens, or a Giotto?

I’m sure there are plenty of art experts who would prefer Rubens or enjoy Giotto more than Da Vinci; for many domains, greatness lies in the eyes of the beholder.

And how does an AI learn in those instances? Well, it largely can’t. Instead, we use a superior model and assume its taste is good, a concept known as LLM-as-a-judge. Let me explain.

One interesting way to deal with domains that aren’t like maths, where the model reaching 4 to the question 2+2 is immediately telling of good or bad, is the use of another model as a judge, a concept also known as ‘distillation’, a term you’ve probably heard multiple times.

These judges not only provide guidance but also help models that get stuck during exploration toward a specific response; without a judge, the model might simply fail to find effective strategies and never actually learn because it never reaches correct responses.

Of course, the question remains how good the responses really are; we are assuming the judge knows what "good” means.

And how do we ensure that? Well, we can’t really mathematically guarantee it, but we can make it likely by:

The first one is self-explanatory and is what people usually call “distillation”, which is basically comparing the student’s responses to how the teacher, or judge, would have responded, “in their shoes.”

This is expensive on a per-word basis because you’re using two models for every prediction, but it effectively shortens training a lot for the student, in the same way that a human student learns faster with tutoring classes, because the teacher helps them find the key strategies faster. This is one of the key components behind China’s rise.

Teacher-forced distillation is fine, but it has one big problem: it’s “off-policy”. The student is still merely copying what the teacher does, not really trying for themselves.

Therefore, it’s the second distillation alternative that researchers are frantically working on.

The reason on-policy learning is powerful is that the student is trained on its own attempts, not on someone else’s perfect answers.

If a teenager studies PhD solutions, they may learn what a brilliant solution looks like, but that does not prove they can produce one. The solution was not generated by their mind. Thus, it may rely on intuitions, shortcuts, and abstractions that they do not yet possess.

A better learning loop is to make the teenager try the problem first, using their current abilities, and only then have the PhD scientist judge the attempt without telling the student how to solve it. Now the feedback lands exactly where it is needed: on the student’s own mistakes, confusions, partial ideas, and almost-correct moves.

In other words, on-policy, having the student learn from its own “thoughts” is a better learning mechanism because it exposes the AI to its own mistakes. It is exploring the space of solutions it can actually reach and then receiving feedback on those solutions.

The idea behind on-policy distillation was popularized by Thinking Machine Labs as a way to enable continual learning. We won’t get into continual learning today, but the idea is to provide a student model with a way to learn from its own responses, using dense teacher feedback; the student still generates the responses entirely on its own, but it receives immediate feedback from the teacher.

This is why it’s the best of both worlds; you aren’t cutting corners by having the student simply imitate generations from the stronger teacher that it would never have generated by itself in that situation, and you aren’t having the student learn entirely by itself, which means it might never learn the right strategies (or will require too much computational effort).

As you can see below, the teacher provides dense feedback for every single word of the student-generated response. While the student is still doing all the work, the teacher is just telling them, ‘uh, this word could be better’ and ‘this one is just fine,’ but without revealing what the student should have done.

That’s how human students really learn, and we’re now applying the same idea to AIs.

However, as outlined earlier, this means we’re running two models for every try, which can be taxing if the teacher is a very large model. So, what if we could use the student to provide their own feedback?

This leads us to on-policy self-distillation.

The idea, which might sound counterintuitive at first, is to use the student as a teacher. But how is that even possible? How can a clueless student serve as its own teacher?

And the answer is very elegant: either training or context.

Say we have the sequence: “What’s the hypotenuse of a right-angle triangle with sides 3 and 4?”

The student, who does know some maths after having imitated the entire history of the human written word, might start trying things out, including Pythagoras’ theorem, but it might take some time to get it right.

As described above, the teacher's role here is not to solve the problem but simply to score the student's attempts, thereby not revealing the answer or the process but ‘guiding’ the model in the correct direction.

To create the teacher, we have two options:

Either way, we prevent needing a larger, smarter, and thus more expensive teacher, and we can use the student to grade itself.

This is far cheaper and thus extremely more scalable than traditional on-policy distillation, and as mentioned, the hottest research avenue by far in AI today.

As of now, you could consider yourself not only on the cusp of the industry in terms of research, but a little bit more of a researcher yourself than you were ten minutes ago.

Now, it’s time to elevate your AI game even further by learning how we’re improving the efficient frontier; how we’re taking these models and making them run much better; or how China might close the gap even further, because if we’re talking about “efficiency,” I don’t have to tell you where the next breakthrough is coming from, right?